When NOT to A/B test: recognizing the wrong tool for the job

A/B testing compares two randomly-split, comparable groups of people. That setup breaks down in a few predictable situations. It's worth recognizing them rather than forcing a test where it doesn't belong.

🎯 Explain Like I'm Hired A/B tests need a fair, random split of comparable people. They stop working when you can't get that split, when the real effect wouldn't show up in a short test window, or when the decision isn't really an optimization question at all. Example: you can't A/B test "should we comply with a new privacy law." There's no version where half your users get the non-compliant experience, and the answer was never going to be decided by which one converts better anyway.