Skip to content

README.md

Latest commit

 

History

History
73 lines (53 loc) · 3.03 KB

README.md

File metadata and controls

73 lines (53 loc) · 3.03 KB

Vanguard A/B Test: UI Statistical Analysis

Authors:
Arunkumar, Supriya & Jahnert, Marc

Project Overview

This project evaluates the impact of a new user interface (UI) on confirmation rates using A/B testing. Our goal is to determine if the UI update led to improved user confirmation rates.

Links

  • Jupyter Notebook: Contains all the code and analyses conducted.
  • Python Functions: Core functions used in the analysis are included in .py files.
  • Tableau Visualizations: Link to Tableau file showcasing data insights.
  • Presentation Slides: Link to Google Slides.
  • Kanban Board: Trello Link for project management.
  • Data Sources: Data Link.

Hypotheses

  1. H1: No significant difference in confirmation rates between Control and Test groups.
  2. H2: Any increase in confirmation rates due to the new UI is not statistically significant at the 5% level.
  3. H3: There is a significant difference in confirmation rates across different age categories.

Methodology

  • Data Cleaning & Merging
  • A/B Testing: Test vs. Control group analysis.
  • Statistical Methods:
    • Two-Proportion Z-Test
    • Chi-Square Test
    • One-Way ANOVA

Key Findings

  • H1 Results:

    • Test Group Confirmation Rate: 14.23%
    • Control Group Confirmation Rate: 12.71%
    • Statistical Significance: Z-statistic: 14.70, P-value < 0.0001. This indicates a significant difference between the confirmation rates of the Test and Control groups.

  • H2 Results:

    • Control Group Confirmation Rate: 56.4%
    • Test Group Confirmation Rate: 66.8%
    • Increase in Confirmation Rates: 10.4%
    • Statistical Significance: Chi-Square Statistic: 220.10, P-value < 0.0001, confirming the increase is statistically significant at the 5% level.

  • H3 Results:

    • F-statistic for Test Group: 43.96, P-value < 0.0001.
    • F-statistic for Control Group: 22.12, P-value < 0.0001.
    • These results show significant differences in confirmation rates across different age categories in both groups.

Conclusion

The new UI led to a significant increase in confirmation rates. Age category also influenced user behavior, suggesting further optimization can be made by targeting specific demographics.

Recommendations

  1. Simplify the User Journey: Reduce the number of steps required for confirmation.
  2. Optimize for Mobile Devices: Ensure the UI performs smoothly on mobile platforms.
  3. Target Age Groups: Consider age-related behaviors when rolling out future updates.

Contact

Feel free to reach out for further details on this project.