Product Analytics with Short Notes 5 #ABTesting
A/B Testing
A/B Test: A statistical hypothesis test by comparing metrics of control and treatment groups to validate their difference.
Steps:
1. Hypothesis
A tentative explanation for an observation, phenomenon, or scientific problem that can be tested by further investigation.
To Mention:
- Evidence available
- Your change
- User segment
- Impact
- Expected KPI Change
2. Control/Treatment Group (Independent Variable)
Control Group gets the product with no treatment, Change not applied.
Treatment Group gets the product with treatment. Change applied.
Practical Tip: At Google Analytics you can divide your users in groups by creating a custom dimension with a small extra script.
3. Metrics to Measure (Dependent Variable)
Primary Metric: North star metric to measure the impact of the change.
Secondary Metric: Other helpful metrics to support your hypothesis.
Health Metrics: Availability, performance, error reports etc.
Binomial Goal Metrics: Binary metrics to store the information if a specific goal is achieved. Not included in the test, but useful to analyze with the results.
4. Run the Experiment
Run the experiment and collect the data. As a rule of thumb run the experiments for at least 1–2 weeks to give users enough time to react.
5. Test the Results
Finally we should test our hypothesis.
Null hypothesis proposes that there is no difference between certain characteristics of a population or data-generating process.
Alternative hypothesis is a direct contradiction of a null hypothesis. This means that if one of the two hypotheses is true, the other is false.If our test results are significant enough to reject the null hypothesis.
We can validate the alternative hypothesis.Which test should we use?:
Measured metric (dependent variable) is …
Continuous — > Significance Test, T-test
Discrete — > Chi-Square Test (Test of Independence)
Resources:
- https://productschool.com/blog/product-management-2/product-management-skills-a-b-testing/
- https://www.youtube.com/watch?v=di2e1QpKYUY
- https://www.investopedia.com/terms/n/null_hypothesis.asp
- https://www.statology.org/chi-square-test-of-independence/
- https://statisticsbyjim.com/hypothesis-testing/t-tests-t-values-t-distributions-probabilities/
