Experiment
One question you’re testing, with a key you reference in code:Control and variant
The two (or more) versions being compared. Control is your current behaviour — the baseline. A variant is a proposed change. Both ship in the same build and run at the same time; different users get different ones. Nobody sees both. There’s no side-by-side comparison — each user gets one version and is unaware the other exists.Traffic split
What share of users lands in each arm, summing to 100. A 50/50 split is the default and the usual choice, since equal groups reach a conclusion fastest.Identity and assignment
Which variant you get is a hash of your identity plus the experiment key. Same identity, same key, same variant — every time, on every device, with no network call. Identity is youruserId once you’ve called identify(), and an anonymous id stored in the
browser before that. Anonymous ids are per browser, so one person on a laptop and a phone is
two participants until they sign in.
Because identity determines assignment, calling identify() can change someone’s
variant — see Variant mismatch.
Exposure
“This user saw this variant.” Fires automatically insidegetVariant().
Exposure is the denominator of your results. If you call getVariant() somewhere the
user never actually sees the thing, you inflate the denominator and dilute your measured
effect.
Conversion
“The thing I care about happened.” The numerator. It only exists where you write it:Conversion rate, and lift
- Absolute — subtract the rates. 18% → 26% is +8 percentage points.
- Relative — the percentage change. The same move is +44%.
Decision metric and guardrail
The decision metric is the conversion event that determines the winner. A guardrail is a second event that catches collateral damage — the case where a variant wins the click but breaks what comes after. Winning the first step while destroying the second is not a win, and the guardrail is what makes that visible.Statistical significance, and mSPRT
A gap between two rates might be real, or it might be noise. Significance is the judgement that it’s too large to be chance. Trevo uses mSPRT, a sequential test, which means results stay valid no matter how often you look. Classical tests require committing to a sample size upfront; checking early inflates false positives. Sequential testing removes that trap.Funnel
An ordered series of events, so you can see which step loses people. Experiments answer “did this change help?”; funnels answer “where are we losing them?”Feature flag
Theif (variant === …) switch itself. In Trevo the flag is temporary by design — when an
experiment concludes, a cleanup PR promotes the winner and deletes the branch along with the
getVariant() call.
Putting it together
1,000 users reached the location gate. 500 saw control, 500 saw the variant — that’s exposure. 90 and 130 respectively got through — that’s conversion. So 18% versus 26%: a +8 point absolute lift, +44% relative. mSPRT says that gap is larger than noise, and the guardrail confirms the recovered users behaved normally afterwards. Ship the variant.