Skip to main content
An experiment produces one number per arm — the conversion rate — and one judgement: whether the difference between them is real.

Conversion rate

Per arm. Raw counts are not the comparison, because arms rarely receive identical traffic and splits are not always even.

Lift

Two ways to express the same movement:
  • Absolute — subtract the rates: 26.0% − 18.0% = +8.0 points
  • Relative — the proportional change: 8.0 ÷ 18.0 = +44%
Both are correct. Relative is the larger, more quotable number; absolute tells you how many additional humans actually converted. When someone reports a big win, check which one they mean — a lift from 0.2% to 0.3% is “+50%” and also almost nothing.

Significance, and why you can peek

A gap between two rates might be a real effect or ordinary randomness. Flip 500 coins twice and you won’t get 250 both times. Classical A/B testing handles this by making you fix a sample size in advance and look only once at the end. Checking early inflates your false-positive rate badly — a dashboard you refresh daily will show you “winners” that evaporate. Trevo uses mSPRT, a sequential test. Its results are always valid: you can look as often as you like, stop as soon as it declares a winner, and the error rate still holds. That fits how people actually use these tools, rather than requiring a discipline nobody sustains. Practically: check whenever you want, and act when Trevo says the result is conclusive rather than when the numbers first look good.

Revenue per user

When the primary event is a purchase carrying value and currency, or the experiment is decided by revenue per user, each arm also shows revenue per user: its revenue divided by everyone in the arm, buyers or not, and paid conversion, the share who paid anything.
An experiment is decided either by conversion rate or by revenue per user, and the choice locks with its conversion events once the experiment moves past approval, when its pull request is generated or it launches. Revenue matters whenever a version could sell more at a lower price: a cheaper plan can win on conversion rate and still earn less. That is why any version that earns significantly less per user is treated like a harmed guardrail. See Paywall and pricing experiments for how amounts are reported, converted and capped. Once people in a test start free trials, revenue counts each person over the same number of days (how long the trials last, plus 3) and only people who entered at least that long ago, and the results say which people that is. While trials have started and no payment reported by your server has followed any, revenue calls no winner: the payment from a trial may simply not be reported yet. In a test of text per language, everyone who reaches the screen counts, including people whose language the new text doesn’t cover and people on builds too old to read it: they see your app’s own text, the same as control. That can only shrink the measured difference, never fake one, and Trevo writes the text in every language it finds in your app so the gap stays small.

Guardrails

The decision metric decides the winner. A guardrail catches the case where winning that metric broke something downstream. Say a more aggressive CTA lifts signup_started by 40%. Real win — unless signup_completed fell, in which case you pulled in people who bounce a screen later and your funnel is worse overall. Pick a guardrail one step after your decision metric, or one that represents quality rather than volume. A result that moves the decision metric and damages the guardrail is not a win, regardless of how good the headline looks.

Inconclusive results are results

Many experiments end with no detectable difference, and that’s normal — it means your change didn’t matter as much as you hoped, which is worth knowing cheaply. Two different endings look similar and aren’t:
  • No effect detected, enough traffic — the change genuinely doesn’t move this metric. Ship whichever version you prefer for other reasons, and move on.
  • Not enough traffic yet — the test cannot yet distinguish a real effect from noise. Keep running, or accept that an effect this small needs more traffic than you have.
Small effects need a lot of users. If your conversion rate is 2% and you’re hoping to find a 10% relative improvement, that’s tens of thousands of exposures per arm. Testing bolder changes is often the faster path than waiting on more traffic.

After a winner

When a result is conclusive, the losing code is dead weight. Ship the winner from the experiment page and Trevo opens a cleanup PR that promotes it into permanent code and removes the variant along with the getVariant() call. Merging that promptly matters more than it sounds: unresolved experiments accumulate into conditional logic nobody remembers the reason for, which is the flag debt this whole workflow exists to avoid.