Sample size & MDE: how long your experiment actually needs to run
The size of the effect you're hunting for decides how many people you need in the test. Hunt for a tiny effect, and you need a lot more people (and time) to find it.
Two questions that go hand in hand: "how small an improvement do I actually care about?" and "how many people do I need in my test to reliably detect it?" The first is called MDE (Minimum Detectable Effect); the second is your sample size.
🎯 Explain Like I'm Hired MDE is the smallest change worth bothering to detect, like deciding you only care about weight changes bigger than half a kilogram, not every single gram. The smaller the effect you want to reliably catch, the more people you need in the test, because tiny effects are easy to lose inside normal random noise unless you have enough data to average it out. Example: detecting a 20% lift in conversion might only need a few thousand people per group; detecting a 1% lift on that same metric might need hundreds of thousands. Same test, wildly different runtime, purely because the effect you're hunting for is smaller.
Sign up to keep reading
Sign up free to unlock the worked examples, edge cases, and interview traps below.