Read this before you plan your first mobile experiment
Three properties of mobile are not choices we made, and every vendor shares them. Planning around them is easier than discovering them mid-experiment. A mobile cohort ramps over weeks, not days. Experiment code ships inside an app release, so your population is whoever has updated. Web reaches full traffic within a day of activation; mobile climbs as adoption climbs, and both variants coexist across app versions for as long as old builds stay installed. Merged is not live. After the experiment PR merges, the experiment page reads Waiting for app release until 20 people on a build with the change have entered it, so store reviewers and test devices don’t count. Store review and updates usually take one to three weeks. The run time, the one-week minimum and the result’s validity checks all count from then, not from the merge. If nobody arrives within four weeks, the page warns that the experiment is not live. Once it is live, the page shows the first build that has it and the share of last week’s active devices on that build or later. An experiment can be switched off without a release, and a later one can change its values. Config is re-read on every foreground, so pausing or stopping is immediate. Changing what a variant does is code, and code ships through store review, unless the code reads a parameter:getParameter('paywall.headline', 'Subscribe') ships once,
and every later experiment just sets new values.
Anonymous users are per-device. A visitor who browses on a laptop and then opens the app
is two participants until they sign in. identify() links them from that point forward;
nothing can link them retroactively. Identifying a user can change their arm because all SDKs
assign the signed-in user from the same user id. Restore auth and call identify() before
showing experiment UI.
Use a Mobile key (tsk_live_…, created under API Keys). A shipped binary can be unpacked,
so a secret key in an app is a leaked key. With a Mobile key, a shipped experiment stays in
the config for a year, so old builds that still have its code keep getting the winner; a Web
key drops it after 30 days.
Durability
Phones lose the network constantly, are killed without warning, and have wrong clocks more often than desktops. Every app SDK is built around that:- Events survive a cold start. The queue is written to device storage and replayed on the next launch. Delivery is at-least-once, and every event carries an idempotency key so a replay is recorded once.
- Timestamps survive a wrong clock. Each batch is stamped with
sentAtwhen it leaves the device. Ingestion compares that against its own clock to recover the device’s offset and correct every event in the batch, which is what lets a queue drained three days late land on a real timeline. Nothing is dropped for being out of range. - Assignment survives no network at all. Config is persisted, so a cold start in a lift assigns the same variant it assigned yesterday instead of falling back to control and silently switching once the network returns.
- A signed-in identity survives a relaunch.
identify()persists the user id, so the next cold start assigns from it rather than reverting to the anonymous id and splitting one person across two identities.
Gating on app version
An experiment whose variant code only exists from a given build onward should not enrol devices that cannot render it. SetminAppVersion on the experiment:
control and record no exposure, so they never appear in the
results. The SDK has to know the build’s version for this: iOS and Android read it from the
app, Flutter and React Native take it as appVersion. Without one, a gated experiment returns
control for everyone, because the SDK cannot prove the build is new enough and guessing wrong
means a broken screen. Every event also records the version, so each event names the build
that sent it.
Store compliance
Apple. The iOS SDK and the React Native package each ship a privacy manifest (PrivacyInfo.xcprivacy), which Xcode aggregates into the app’s. It declares product
interaction and a user id, both linked, neither used for tracking, and no required-reason
APIs. The Flutter SDK is pure Dart with no native code, so it ships none: declare the same two
data types in your app’s own manifest.
Google Play. Data safety answers you can paste into the Play Console:
The anonymous id is minted on device and is not an advertising or hardware identifier, which
is why “Device or other IDs” is No. Declaring it there invites a policy review you do not
need.