An appropriate sample size

Before you run an A/B test, it’s important to make sure your sample size is large enough. If your sample is too small, you might not be able to tell whether the difference you see between groups is real or just due to random chance.

Let’s take a coin flip as an example. If I flipped a quarter 3 times and got heads twice, I might think heads comes up more often than tails. But with such a small number of flips, that result could easily be a fluke. Now imagine I flipped the coin 1,000 times and got 520 heads — I’d feel much more confident saying the coin is probably fair (or close to it).

A/B tests work the same way. The more people or events included in the test, the more confident we can be in the results. In fact, the statistical test we use will output a number — often called a p-value or confidence level — that tells us how likely it is that the difference between groups is real. This number is heavily influenced by how much data we collect.

That’s why planning for sample size is so important. If we want to be very confident in our results, we’ll need a large enough sample. But in real-world settings, there’s often a trade-off: running a longer test gives us more certainty, while shorter tests might give faster answers (and quicker product changes) but with more uncertainty.

Reasonably identical features along A/B

For an A/B test to be fair, the control group and the experiment group should be similar in every important way except for the thing you’re testing.

Imagine testing a new medication designed to help people with a specific illness. It wouldn’t make sense to include people without that illness in the experiment group but only people with the illness in the control group. That would make it impossible to know if differences in results were due to the medicine or just differences in who was in each group.

Similarly, if there’s another health condition that could affect how well the medicine works, you’d want to make sure that condition is either equally represented in both groups or excluded altogether. If not, the results could be misleading.

This idea applies to any A/B test: the groups should be as similar as possible so you can be confident that differences in outcomes are caused by the change you made — not other factors.

Random Sampling

To make sure the control and experiment groups are as similar as possible, we assign users to each group using random sampling.

Random sampling means each user has an equal chance of being placed in either group. This helps avoid bias — so the groups don’t differ in any unexpected ways that could affect the results.

By randomly assigning users, we can be more confident that any differences we see are due to the change we’re testing, not other factors.

The Student’s T-Test

A very common type of statistical test run in an A/B experiment design is the Student’s T-test (published by a statistician and brewery chemist using the pseudonym student). This is used to compare the mean value of a variable of interest across our groups to determine if there is a statistically significant difference. I.e., is the average weight between this group that took a new weight loss pill and their placebo control group different and is that difference meaningful or just random chance? Note that this is not the same question as does this drug cause weight loss? A/B tests do NOT imply causality.

There are a few different types of t-test depending (one sample, two sample, paired), but without going into these I’ll just say two sample tests are what are used in A/B testing (two groups = two samples).

Z-test

Similar to the Student’s T-Test, however this measures difference in a binary value. I.e., does this product change yield more customer sign-ups than the status-quo. This has similar one sample, two sample, paired flavors as the t-test.

Other Tests

These are the two primary tests that get utilized in A/B testing. There are a myriad of other statistical tests that fit different purposes in particular scenarios. Non-parametric counterparts to the t and z-test (Mann-Whitney U and Chi-square tests, respectively) can be used in lieu of the t and z-test when the assumption of normality is not met. However they compare sample medians or ranks, where as the t and z-test compare means or proportions.

There are also tests that extend beyond the A/B structure. ANOVA (analysis of variance) can be used to detect if there is some statistically significant variance among any number of groups, not just two.

Forming a Hypothesis

All inferential statistical tests begin with a hypothesis. For an A/B test we hope or hypothesize that there will be some measurable difference in outcome between two groups.

So let me throw out some arbitrary hypothesis.

$$H_0: Women\ and\ men\ are\ equally\ smart\ (no\ difference\ in\ exam\_scores) \\ H_{a}: Women\ are\ smarter\ than\ men\ (will\ have\ higher\ exam\_scores )$$

The Mechanics of the Test

Assuming you know how to calculate an average and a standard deviation, this is the formula to get the t-statistic that will allow us to compare the means of our two groups.

With the value t we do further work to compare against a critical value we obtain from a t-distribution, but going into that is a little further than I want to write about. Just let it be known we can calculate this by hand given some resources - or we can use statistical packages in R to quickly output the results.

# seed for reproducability
set.seed(6391)

# work has been done separately to ensure there are no confounding variables

# sampling 5000 rows of women
rando_women <- grades_and_features %>%
  filter(gender == 'Female') %>%
  .[sample(nrow(.), 5000), ] %>%
  pull(exam_score)
    
# sampling 5000 rows of men
rando_men <- grades_and_features %>%
  filter(gender == 'Male') %>%
  .[sample(nrow(.), 5000), ] %>%
  pull(exam_score)

t.test(rando_men, rando_women)

## 
## 	Welch Two Sample t-test
## 
## data:  rando_men and rando_women
## t = 0.25821, df = 9998, p-value = 0.7962
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.3994445  0.5206445
## sample estimates:
## mean of x mean of y 
##   89.0694   89.0088

The p-value tells us how likely it is that the difference in means we observed could happen just by chance if there were no real difference in the population. A p-value of 0.7962 means the observed difference is not statistically significant. Likewise, we can take a look at the means of both groups and see how they are nearly identical. In this case we do not have enough evidence to reject the null hypothesis and must accept that women and men have roughly the same exam scores.

As a Tool for Feature Selection.

Now consider beyond testing various groups for meaningful differences in exam scores, that I was even going so far as to develop to formula to help predict exam score. I could make a linear regression model, or some derivation of this model type, but then how would I go about choosing what features make the most sense? We’ve already seen there’s no meaningful difference in exam scores based on gender. I guess I could just continue applying some inferential test along all the categorical variables in my data, but that could take a long time and would not help me choose amongst the numeric variables.