Vanguard A/B Test: UI Statistical Analysis

Authors:
Arunkumar, Supriya & Jahnert, Marc

---

Project Overview

This project evaluates the impact of a new user interface (UI) on confirmation rates using A/B testing. Our goal is to determine if the UI update led to improved user confirmation rates.

---

Links

• Jupyter Notebook: Contains all the code and analyses conducted.
• Python Functions: Core functions used in the analysis are included in .py files.
• Tableau Visualizations: Link to Tableau file showcasing data insights.
• Presentation Slides: Link to Google Slides.
• Kanban Board: Trello Link for project management.
• Data Sources: Data Link.

---

Hypotheses

1. H1: No significant difference in confirmation rates between Control and Test groups.
2. H2: Any increase in confirmation rates due to the new UI is not statistically significant at the 5% level.
3. H3: There is a significant difference in confirmation rates across different age categories.

---

Methodology

• Data Cleaning & Merging
• A/B Testing: Test vs. Control group analysis.
• Statistical Methods:
  • Two-Proportion Z-Test
  • Chi-Square Test
  • One-Way ANOVA

---

Key Findings

• H1 Results:

  • Test Group Confirmation Rate: 14.23%
  • Control Group Confirmation Rate: 12.71%
  • Statistical Significance: Z-statistic: 14.70, P-value < 0.0001. This indicates a significant difference between the confirmation rates of the Test and Control groups.

• H2 Results:

  • Control Group Confirmation Rate: 56.4%
  • Test Group Confirmation Rate: 66.8%
  • Increase in Confirmation Rates: 10.4%
  • Statistical Significance: Chi-Square Statistic: 220.10, P-value < 0.0001, confirming the increase is statistically significant at the 5% level.

• H3 Results:

  • F-statistic for Test Group: 43.96, P-value < 0.0001.
  • F-statistic for Control Group: 22.12, P-value < 0.0001.
  • These results show significant differences in confirmation rates across different age categories in both groups.

---

Conclusion

The new UI led to a significant increase in confirmation rates. Age category also influenced user behavior, suggesting further optimization can be made by targeting specific demographics.

---

Recommendations

1. Simplify the User Journey: Reduce the number of steps required for confirmation.
2. Optimize for Mobile Devices: Ensure the UI performs smoothly on mobile platforms.
3. Target Age Groups: Consider age-related behaviors when rolling out future updates.

---

Contact

Feel free to reach out for further details on this project.