Experimentation
A/B testing math, MDE, p-values, and political reality.
Reading an experiment readout: five checks before you trust it
A readout with a green checkmark and a big lift number is exactly the moment to slow down, not speed up. Here's what to check before believing it.
P-values: the number everyone quotes and almost everyone misreads
A p-value is not 'the chance your result is a fluke.' It's a narrower, stranger question, and misreading it is the single most common statistics mistake in the industry.
Novelty & primacy effects: why an early test result might not last
A change can look like a huge win in week one and fade by week three, or start weak and grow stronger. Both are predictable, named patterns, not random noise.
When NOT to A/B test: recognizing the wrong tool for the job
A/B testing is the default reflex in analytics. Some decisions genuinely can't be tested that way, and forcing one wastes time or actively misleads.
Hypothesis design: turning a hunch into something you can actually test
A vague hope like 'let's improve the product page' can't be tested. A real hypothesis names the exact change, the metric, and why you expect it to work.
Sample size & MDE: how long your experiment actually needs to run
The size of the effect you're hunting for decides how many people you need in the test. Hunt for a tiny effect, and you need a lot more people (and time) to find it.