How to Choose the Right Sample Size (Without Overcomplicating It)

If you’ve ever run an A/B test and found yourself asking, “How many users do I need?”—you’re not alone. Sample size calculations are one of the most misunderstood (and often overlooked) aspects of A/B testing. But here’s the truth: getting the sample size wrong can lead to bad decisions, wasted time, and misleading insights.

Today, we’re going deep into how to calculate the right sample size—without spreadsheets or headaches—and why this step is crucial for the success of your experiments.

Why Sample Size Matters

Running a test with too few users means your results could be statistically unreliable—you might see differences that aren’t real or miss changes that are. On the flip side, running a test with way too many users waste resources and delays decision-making.

A properly chosen sample size ensures:

1. Statistical significance: Your results aren’t due to random chance.

2. Practical relevance: You detect meaningful differences that matter to your business.

3. Efficient testing: You don’t waste resources collecting unnecessary data.

The 4 Inputs You Need for Sample Size Calculation

Before diving into the formula, let’s simplify things. All you need are these four inputs:

1. Baseline conversion rate
   What’s the current performance of the metric you’re testing (e.g., click-through rate, purchase rate)?

   • Example: If 10 out of 100 users typically convert, your baseline is 10%.

2. Minimum detectable effect (MDE)
   What’s the smallest improvement that would make this test worth it?

   • Example: If you’re testing a call-to-action button and only care about changes of at least 5%, your MDE is 5%.

3. Statistical significance level (alpha)
   How confident do you want to be that your results aren’t due to random chance? Most A/B tests use a 5% significance level (95% confidence).

4. Statistical power (1-beta)
   How confident do you want to be in detecting a true difference? The standard is 80% power, which means there’s a 20% chance of missing a real effect.

Simplifying the Formula

The full formula for calculating sample size is intimidating—and unnecessary for most decision-makers. Instead, here’s a simpler way to estimate:

Subscribe to our premium content to read the rest.