P-values: the number everyone quotes and almost everyone misreads
A p-value is not 'the chance your result is a fluke.' It's a narrower, stranger question, and misreading it is the single most common statistics mistake in the industry.
A p-value is a number that comes out of almost every A/B test, and it's widely misunderstood, even by people who use it every day. Getting this right is worth slowing down for.
🎯 Explain Like I'm Hired A p-value answers one narrow question: "if there were actually no real difference between my two versions, how likely would I be to see a gap this big just from random luck?" A small p-value means "pretty unlikely by luck alone," which is evidence against "no difference," but it says nothing about how big the real difference is, or how likely your idea is to be correct. Example: p = 0.03 means "if the button color truly made no difference at all, there was only a 3% chance of seeing a gap this large by random noise." It does not mean "there's a 97% chance the new button is better." That's a different question the p-value can't answer.
Sign up to keep reading
Sign up free to unlock the worked examples, edge cases, and interview traps below.