Reading an A/B readout

An A/B test result is a small, expensive promise. Most readouts are read too quickly and trusted too much. The fix is four questions you ask before you read the number, not after.

The four questions

• Was the exposure large enough to see the effect you claim?
• Did the variant reach the population you said it would?
• Is the metric you are reading the metric you designed for?
• What is the cost of being wrong, and in which direction?

The first two are about validity. The third is about honesty — it is easy to read a metric that moved instead of the one you planned for. The fourth is about stakes. A 1% lift on a low-stakes metric is not the same as a 1% lift on a high-stakes one.

A readout you can defend is worth ten you can only celebrate.

In the cohort we run a live readout with the group and grade it against these four questions. It is the single exercise that changes how people read results afterwards.