Vanguard A/B Testing Analysis

UI Redesign Experiment for Conversion Optimisation

Author: Marisa Oliveira Project Type: Data Analytics / A/B Testing Year: 2026

Developed as part of a group project. My contribution focused on data cleaning, exploratory analysis, KPI design, and statistical testing.

---

🚀 Key Results

• Increased completion rate from 49.9% → 58.6% (+8.7pp lift)

• Confirmed statistical significance (p-value < 0.05)

• Identified increased friction despite higher conversion:

  • Backtracks: 0.30 → 0.44
  • Error rate: 8.8% → 11.6%

• Demonstrated that improved UX can increase conversion while introducing cognitive load

---

📌 Project Overview

This project evaluates a digital A/B test conducted by Vanguard to determine whether a redesigned user interface improves the completion rate of an online investment process.

The redesigned UI introduced in-context prompts (hints, cues, guidance) aimed at improving usability and increasing successful task completion.

---

🧠 Business Context

The experiment was designed to improve digital adoption and increase successful investment actions, a key driver of assets under management (AUM).

The core question:

Does a more intuitive interface lead to higher completion rates?

---

🧪 Experiment Design

• Control Group: Traditional interface
• Test Group: Modern UI with in-context prompts
• Funnel Steps: start → step_1 → step_2 → step_3 → confirm

---

📊 Key Metrics

• Completion Rate (Primary KPI)
• Backtracks (user friction indicator)
• Error Rate (non-linear navigation)
• Session Duration

---

📈 Key Insights

Conversion Impact

The redesigned UI significantly improved completion rates:

• Control: ~49.9%
• Test: ~58.6%

This represents a meaningful uplift in conversion.

---

Friction Trade-off

Despite higher conversion:

• Users in the Test group showed more backtracking
• Error rates increased
• Time spent on the final step increased significantly

👉 Insight: Improved UX does not always mean reduced friction — users may engage more deeply or double-check decisions.

---

Statistical Validation

A Z-test for proportions confirmed that the uplift in completion rate is statistically significant:

• p-value < 0.05
• Reject null hypothesis
• UI redesign is likely the driver of improvement

---

💰 Business Impact

• Conversion uplift exceeded the 5% threshold required for rollout
• Enables higher AUM through increased successful transactions
• Provides evidence-based justification for UI redesign

---

🧩 Challenges & Approach

Data Challenges

• Converting non-linear clickstream data into a structured funnel
• Detecting backtracking behaviour
• Cleaning timestamps to measure true session duration

Analytical Approach

• Funnel reconstruction from event logs
• Feature engineering for behavioural metrics
• Hypothesis testing using statistical methods

---

🎯 Recommendations

• Roll out the new UI based on conversion uplift
• Optimise the confirm step to reduce friction and time spent
• Continue monitoring long-term customer value (AUM, retention)

---

🛠 Tools & Technologies

• Python (Pandas, NumPy, SciPy)
• Tableau (visualisation)
• Jupyter Notebook
• Git & GitHub

---

📁 Project Structure

• Data cleaning & exploration notebook
• Experiment analysis notebook
• Processed datasets
• Documentation

---