Conversion rate
Lift
Two ways to express the same movement:- Absolute — subtract the rates:
26.0% − 18.0%= +8.0 points - Relative — the proportional change:
8.0 ÷ 18.0= +44%
Significance, and why you can peek
A gap between two rates might be a real effect or ordinary randomness. Flip 500 coins twice and you won’t get 250 both times. Classical A/B testing handles this by making you fix a sample size in advance and look only once at the end. Checking early inflates your false-positive rate badly — a dashboard you refresh daily will show you “winners” that evaporate. Trevo uses mSPRT, a sequential test. Its results are always valid: you can look as often as you like, stop as soon as it declares a winner, and the error rate still holds. That fits how people actually use these tools, rather than requiring a discipline nobody sustains. Practically: check whenever you want, and act when Trevo says the result is conclusive rather than when the numbers first look good.Revenue per user
When the primary event is apurchase carrying value and currency, or the experiment is
decided by revenue per user, each arm also shows revenue per user: its revenue divided by
everyone in the arm, buyers or not, and paid conversion, the share who paid anything.
Guardrails
The decision metric decides the winner. A guardrail catches the case where winning that metric broke something downstream. Say a more aggressive CTA liftssignup_started by 40%. Real win — unless
signup_completed fell, in which case you pulled in people who bounce a screen later and
your funnel is worse overall.
Pick a guardrail one step after your decision metric, or one that represents quality rather
than volume. A result that moves the decision metric and damages the guardrail is not a win,
regardless of how good the headline looks.
Inconclusive results are results
Many experiments end with no detectable difference, and that’s normal — it means your change didn’t matter as much as you hoped, which is worth knowing cheaply. Two different endings look similar and aren’t:- No effect detected, enough traffic — the change genuinely doesn’t move this metric. Ship whichever version you prefer for other reasons, and move on.
- Not enough traffic yet — the test cannot yet distinguish a real effect from noise. Keep running, or accept that an effect this small needs more traffic than you have.
After a winner
When a result is conclusive, the losing code is dead weight. Ship the winner from the experiment page and Trevo opens a cleanup PR that promotes it into permanent code and removes the variant along with thegetVariant() call.
Merging that promptly matters more than it sounds: unresolved experiments accumulate into
conditional logic nobody remembers the reason for, which is the flag debt this whole workflow
exists to avoid.