Why architecture matters here
Experimentation matters architecturally because the thing it produces — a decision to ship or not — is only as trustworthy as the weakest link in the chain that produced it, and every link is a place where a subtle bug produces a confident wrong answer. This is different from most systems, where a bug produces an error you can see. Here a bug produces a number, and the number looks fine. A skewed assignment, a lost exposure log, an analysis that peeked — none of these throw exceptions; they just shift the estimate, and the team ships the wrong thing believing the data told them to. The architecture exists to make each link verifiable so that trust in the final number is earned rather than assumed.
Deterministic assignment is the first foundation. The same unit — user, account, device, session, depending on what you are randomizing — must map to the same variant every time it is evaluated, on every server, for the life of the experiment. The standard technique is to hash the unit id together with an experiment-specific salt and take the result modulo the bucket space: bucket = hash(unit_id + salt) % N. Because the hash is pure, no shared state or coordination is needed; a thousand servers independently compute the same bucket for the same user. Determinism is what makes a user's experience stable and their behaviour attributable to a single variant, which is the precondition for the whole measurement to mean anything.
Honest exposure logging is the second foundation, and the one most often botched. An experiment measures the effect of a treatment on the users who actually encountered it. If you analyze everyone who was eligible rather than everyone who was exposed, you dilute the effect with users who never reached the changed surface — a checkout experiment analyzed over all users, not just those who reached checkout, buries a real effect under noise. The exposure log must be written at the exact call site where the variant is resolved and the user first sees the difference, capturing unit, experiment, variant, and timestamp. That record is the ground truth the analysis joins against; get it wrong and no amount of statistical sophistication recovers the truth.
The statistics are the third foundation, and their architectural implication is that analysis must be gated, not ad hoc. Two classic errors dominate. Peeking: checking results continuously and stopping the moment p-value dips below 0.05 inflates the false-positive rate far above the nominal 5%, because with enough looks the random walk of the estimate will cross the threshold by chance. Multiple comparisons: testing twenty metrics and celebrating the one that came up significant ignores that one-in-twenty will do so by luck. The platform defends against both by committing to a fixed sample size or a valid sequential test up front, designating a primary metric, and correcting for the secondary ones — turning statistics from a thing analysts do into a thing the system enforces.
Finally, experiments do not run alone, and that is why governance is foundational rather than optional. At any moment a product may have hundreds of live experiments, and without coordination they interfere: two experiments both changing the checkout button confound each other, or a user lands in a combination of treatments no one intended. The registry's exclusion and mutual-exclusion groups, its ramp schedules, and its ownership records are what let many experiments share the same users safely — by ensuring conflicting ones never overlap and every result traces back to a stated hypothesis and an accountable owner. A platform that randomizes correctly but cannot compose experiments will collapse under its own success the moment more than a handful run at once.
The architecture: every piece explained
Start at the top row. The experiment config is the declaration: an experiment key, the variants and their allocation percentages (say 50/50, or 90/10 during a cautious ramp), the targeting rules for who is eligible, and the randomization unit. The assignment service turns that config plus a unit id into a variant by hashing — hash(unit_id + experiment_salt) mapped into the allocation — so assignment is deterministic, uniform, and independent across experiments because each has its own salt. Independence across salts is what stops the same users from always landing in the 'A' side of every experiment, which would systematically bias the whole program.
The SDK / feature gate is where assignment meets the product. At the call site — the code path that renders the button, chooses the ranking model, sets the timeout — the SDK asks 'which variant is this unit in for experiment X?' and branches. Critically, the same SDK call that resolves the variant emits the exposure log: a record that this unit was bucketed into this variant at this instant. Co-locating resolution and logging is deliberate — it guarantees you log exactly the users who were actually exposed, no more (eligible-but-never-reached users) and no less (exposed-but-forgotten users).
The middle row is measurement. The event stream carries the product's ordinary telemetry: clicks, purchases, latencies, errors, revenue — the raw events from which metrics are built. The metrics pipeline performs the central join: for each unit in the exposure log, gather its events within the experiment window and aggregate them into the metrics of interest, attributed to the unit's variant. This join is the heart of the analysis; it converts 'user 42 saw variant B and later bought something' into 'variant B's conversion rate.' The stats engine then takes the per-variant metric distributions and computes the treatment effect, a confidence interval, and a p-value or Bayesian posterior — quantifying not just whether B beat A but by how much and how sure we are.
The bottom row is what separates a toy from a platform. Guardrail monitors watch for the failures that invalidate an experiment regardless of its headline result. The most important is the sample-ratio mismatch (SRM) check: if you allocated 50/50 but observe 52/48 with millions of users, the split is statistically impossible by chance, which means assignment or logging is broken and every metric from the experiment is suspect. Other guardrails watch latency, error rate, and crash rate so a treatment that wins on the primary metric but tanks reliability is caught before it ships. The decision surface presents the effect, its interval, and the guardrail status together, so the ship/hold/roll-back call is made on the full picture. The registry underneath ties every experiment to a hypothesis, an owner, a ramp schedule, and its exclusion groups, so the whole population of experiments stays governed and composable rather than a free-for-all.
Two properties cut across the design. First, the unit of randomization must match the unit of analysis: if you randomize by user you must analyze by user, not by session, or you violate the independence the statistics assume and understate your uncertainty. Second, exposure is the spine: config, assignment, and events all exist to feed and be joined against the exposure log, and its integrity is the single most load-bearing thing in the system. Every other component can be sophisticated, but if the exposure log is wrong, the platform is confidently lying.
End-to-end flow
Follow one experiment from idea to decision. A designer hypothesizes that a simpler checkout button will lift completed purchases. They create an experiment in the registry: a stated hypothesis, a primary metric (checkout completion rate), guardrail metrics (latency, payment-error rate), a randomization unit (user), a 50/50 allocation, and a mutual-exclusion group so no other checkout experiment can overlap. The registry checks the exclusion groups, confirms no conflict, and the experiment is provisioned but not yet ramped.
The team ramps to 5% first — a safety valve. The assignment service now buckets 5% of users into the experiment, splitting them 50/50 between the old and new button. When one of those users reaches checkout, the SDK resolves their variant and writes an exposure record. Over the first hours, the guardrail monitors watch the SRM check and the error/latency guardrails. Because the new button is fine technically, guardrails stay green, so the team ramps to the full 50%. The 5% ramp caught nothing this time, but it is the step that would have caught a treatment that crashed on some devices before it reached everyone.
Now the measurement machinery runs continuously but the decision waits. Each user who checks out generates events; the metrics pipeline joins those events to the exposure log, attributing each purchase to the user's variant, and the stats engine updates the estimated lift and its confidence interval. The platform does not let the team stop the instant the interval excludes zero — it enforces the pre-committed sample size (or a valid sequential boundary), because stopping on the first favorable peek is exactly the peeking error that manufactures false positives. The dashboard shows the estimate accumulating but labels the result 'not yet decisive' until the design's stopping rule is satisfied.
The experiment reaches its planned sample size. The stats engine reports the new button lifted completion by 1.8% with a 95% confidence interval of [0.9%, 2.7%] — the interval excludes zero, so the effect is significant, and its magnitude is product-meaningful. The SRM check confirms the observed split is 49.98/50.02, consistent with the intended 50/50, so assignment and logging are trustworthy. Guardrails are green: latency and payment errors were unchanged. The decision surface shows all of this together, and the team ships the new button to 100%, retiring the experiment in the registry with its result recorded against the original hypothesis.
Contrast the path where something is wrong. Suppose the SRM check had flagged a 51/49 split at high volume. The stats engine might still show a significant lift, but the platform blocks the decision: an impossible split means some users are being misassigned or their exposures dropped, so the metric — however pretty — cannot be trusted. The team investigates, finds that a caching bug caused some treatment users to be logged as control, fixes it, and reruns. The value of the architecture is precisely this refusal: it would rather withhold a decision than let a corrupted experiment produce a confident, wrong ship.