Skip to main content
An experiment produces one number per arm — the conversion rate — and one judgement: whether the difference between them is real.

​
Conversion rate

Per arm. Raw counts are not the comparison, because arms rarely receive identical traffic and splits are not always even.

​
Lift

Two ways to express the same movement:
  • Absolute — subtract the rates: 26.0% − 18.0% = +8.0 points
  • Relative — the proportional change: 8.0 ÷ 18.0 = +44%
Both are correct. Relative is the larger, more quotable number; absolute tells you how many additional humans actually converted. When someone reports a big win, check which one they mean — a lift from 0.2% to 0.3% is “+50%” and also almost nothing.

​
Significance, and why you can peek

A gap between two rates might be a real effect or ordinary randomness. Flip 500 coins twice and you won’t get 250 both times. Classical A/B testing handles this by making you fix a sample size in advance and look only once at the end. Checking early inflates your false-positive rate badly — a dashboard you refresh daily will show you “winners” that evaporate. Trevo uses mSPRT, a sequential test. Its results are always valid: you can look as often as you like, stop as soon as it declares a winner, and the error rate still holds. That fits how people actually use these tools, rather than requiring a discipline nobody sustains. Practically: check whenever you want, and act when Trevo says the result is conclusive rather than when the numbers first look good.

​
Guardrails

The decision metric decides the winner. A guardrail catches the case where winning that metric broke something downstream. Say a more aggressive CTA lifts signup_started by 40%. Real win — unless signup_completed fell, in which case you pulled in people who bounce a screen later and your funnel is worse overall. Pick a guardrail one step after your decision metric, or one that represents quality rather than volume. A result that moves the decision metric and damages the guardrail is not a win, regardless of how good the headline looks.

​
Inconclusive results are results

Many experiments end with no detectable difference, and that’s normal — it means your change didn’t matter as much as you hoped, which is worth knowing cheaply. Two different endings look similar and aren’t:
  • No effect detected, enough traffic — the change genuinely doesn’t move this metric. Ship whichever version you prefer for other reasons, and move on.
  • Not enough traffic yet — the test cannot yet distinguish a real effect from noise. Keep running, or accept that an effect this small needs more traffic than you have.
Small effects need a lot of users. If your conversion rate is 2% and you’re hoping to find a 10% relative improvement, that’s tens of thousands of exposures per arm. Testing bolder changes is often the faster path than waiting on more traffic.

​
After a winner

When a result is conclusive, the losing code is dead weight. Trevo opens a cleanup PR that promotes the winner into permanent code and removes the variant along with the getVariant() call. Merging that promptly matters more than it sounds: unresolved experiments accumulate into conditional logic nobody remembers the reason for, which is the flag debt this whole workflow exists to avoid.