A/B Test Analyzer

Business problem

Product teams need a defensible way to decide whether an experiment should ship, keep running, or stop because the data is too weak.

Dataset and context

Illustrative landing-page A/B test dataset with 100 rows split across control and treatment groups. The app is designed as an analyst workflow template: validate input, calculate effects, run tests, check uncertainty, and explain rollout risk.

Methodology

• -Validated experiment inputs, assignment groups, conversion columns, and numeric outcome fields.
• -Calculated conversion lift, time-on-page difference, confidence intervals, and p-values.
• -Used frequentist tests for primary readout and Bayesian beta-binomial checks as a second lens.
• -Included power analysis and guardrail language so the answer is not reduced to a single p-value.
• -Packaged the workflow in Streamlit with CI checks for the underlying analysis code.

Technical approach

The project separates analysis logic from the Streamlit interface. Statistical functions can be tested directly, while the app layer focuses on upload, metric selection, and interpretation. This makes the project easier to inspect in GitHub and safer to extend.

Key insights

• -Conversion increased from 42.0% to 66.0%, a +24.0 percentage-point lift.
• -The conversion result is statistically significant with p=0.008026.
• -Time-on-page increased by 1.691 minutes with p=0.000139.
• -The README correctly notes that the small sample is illustrative and should not be oversold.

Business recommendations

• -Use the tool as a repeatable first-pass experiment readout for product and growth teams.
• -Add revenue, retention, and support guardrails before treating conversion lift as a rollout decision.
• -Require pre-test power planning for high-stakes experiments.
• -Keep Bayesian checks as a communication layer for stakeholders who need probability-style language.

Results and impact

The experiment dataset is illustrative. The value of the project is the analysis workflow, test coverage, interpretation, and deployable analyst tool.

What I would improve in production

• -Add sample-ratio mismatch checks and automated guardrail alerts.
• -Support sequential testing rules for experiments monitored during rollout.
• -Add CUPED or pre-period covariate adjustment when pre-experiment behavior is available.
• -Store experiment readouts so teams can compare historical decisions and outcomes.

Recruiter takeaway

This project shows product analytics judgment: the app gives a clear answer, but it also exposes uncertainty, sample-size limits, and rollout guardrails.