Why architecture matters here
Consider what teams do without a flag platform. Releases couple to deploys, so risky changes wait for release windows and the windows grow rituals. Rollback means redeploying the previous build — fifteen minutes of exposure, minimum, for a defect that a flag would have extinguished in seconds. Gradual rollout means deploying to a canary fleet, which segments by instance when the risk usually segments by user. A/B testing gets hand-rolled per team with subtly different bucketing, producing experiment results nobody quite trusts. Each of these is a workaround for the missing primitive: runtime, per-user control over which code path executes.
The primitive is only trustworthy if its architecture holds under three pressures. Latency: a checkout page might evaluate thirty flags; at one RPC each, the flag system would dominate the page budget, so evaluation must be an in-memory lookup measured in microseconds. Freshness: a kill switch that takes ten minutes to propagate is not a kill switch; rule changes must reach every SDK in seconds, globally. Availability: the flag service will have its own outages, and the failure semantics must be keep serving the last-known rules, never block the request or flip everyone to default — the latter being a silent mass-unlaunch, one of the most damaging incidents a platform can cause.
And because flags gate behavior, they accumulate organizational weight: compliance wants an audit trail of who enabled what for whom; experimentation wants exposure logs tied to consistent assignment; finance wants to know which kill switches protect revenue. The architecture below is what makes all of that a byproduct of normal operation instead of a reconstruction project.
The counterfactual is worth naming: teams without a real flag platform still flag things — with environment variables, database rows read at startup, if-statements keyed on org lists baked into code. Each is a flag system missing one load-bearing property: env vars need restarts, startup reads need redeploys to change, hardcoded lists need code review to amend at 3 a.m. The organization pays flag-platform complexity either way; the only choice is whether it pays as one coherent system with propagation guarantees and an audit trail, or as forty ad-hoc reinventions, each discovered during a different incident.
The architecture: every piece explained
The control plane. Flags, variants, targeting rules, and environment scoping live in a database behind an API and UI with review workflows and a full audit log. A rule is an ordered list of clauses — users in segment beta-testers: variant on; users in DE: variant off; everyone else: 5% on by hash — evaluated top-down, first match wins. The control plane compiles each change into a versioned, immutable ruleset snapshot: a compact document containing every flag for an environment, monotonically numbered so consumers can detect staleness and skew.
Delivery. Snapshots reach SDKs by two complementary paths. A streaming channel — SSE or long-poll against a fanout tier — pushes deltas within a second or two of publication; this is what makes kill switches real. A CDN path serves full snapshots for bootstrap, so a cold-starting pod gets its ruleset in one cached GET without stampeding the origin. SDKs persist the last snapshot to local disk or a sidecar cache, giving them a last-known-good to serve through any control-plane outage, and every flag call site supplies a code-level default as the final floor.
Evaluation and bucketing. Evaluation happens in-process: rules run against a context object (user id, org, region, platform) in microseconds. Percentage rollouts hash unit_id + flag_salt into a bucket space of, say, 10,000 slots; the salt-per-flag decorrelates experiments so the same 5% of users is not the treatment group for everything. Determinism is the contract: any SDK in any language, given the same context and ruleset version, returns the same variant — which is what lets a web frontend, a mobile client, and three backend services agree about one user mid-request.
Exposure and lifecycle. Each evaluation optionally emits an exposure event — flag, variant, ruleset version, unit id — batched to the analytics pipeline, joining assignments to outcome metrics for experiment readouts. Around the whole thing sits lifecycle machinery: every flag has an owner and an intended lifespan (release flag, experiment, permanent operational switch), expiry reminders, and a cleanup path that turns a 100%-rolled-out flag into deleted code within a sprint or two.
Client-side surfaces are a different regime. Backend SDKs hold the full ruleset and evaluate any user; shipping that ruleset to a browser or mobile app would leak targeting logic — segment names, percentage ramps, competitor org IDs — to anyone with devtools. Client SDKs therefore evaluate server-side per user: the app sends its context to an evaluation endpoint (or edge worker) and receives only the resolved variants for that one user, cached locally with a short TTL and a streaming channel for mid-session updates. Mobile adds offline reality: variants persist across launches so the app boots with its last assignments rather than defaults on a subway. The variant payloads are also where remote-config-style values live — a JSON blob per variant — which keeps copy tweaks and threshold changes out of the release train entirely.
End-to-end flow
A payments team merges a new fraud-scoring model behind fraud_model_v2, default off, owner set, expiry ninety days out. The code deploys to production on Thursday; nothing changes, because the flag evaluates off everywhere — deploy and release have been decoupled. Monday 09:30, the team enables the flag for the internal-employees segment via a reviewed control-plane change. The compiler publishes snapshot v18,241; the fanout tier streams the delta, and within two seconds every SDK across 300 pods is evaluating the new rule. Employees start hitting the v2 model; exposure events flow into the pipeline tagged with the ruleset version.
Wednesday, satisfied with internal results, the team moves to a 5% percentage rollout. Bucketing hashes each user id with the flag salt; user 48,291 lands in bucket 312 of 10,000, inside the 0-499 range — she gets v2 consistently, on the web checkout, the mobile app, and the backend risk service, because all three evaluate the same hash with the same salt. The experiment dashboard accumulates: fraud catch rate up, false-positive guardrail flat, checkout latency p95 unchanged. Rollout steps to 25%, then 50%.
Thursday 14:12, the on-call sees payment-decline alerts for one card network. First move in the runbook: flip the fraud_model_v2 kill switch — a single control-plane action, no deploy, no approval chain at severity-1. Snapshot v18,299 propagates in 1.8 s; declines recover within the minute. The postmortem uses exposure events joined against decline logs to confirm v2 mishandled one issuer response code, the fix ships dark on Friday, and the rollout resumes Monday from 50% — no user saw a redeploy, and the experiment analysis simply excludes the incident window by ruleset version. Ninety days later, v2 at 100% and stable, the expiry bot opens the cleanup PR that deletes the flag and the v1 code path.
The exposure pipeline earns its keep once more during the rollout. At the 25% step, the sample-ratio-mismatch monitor flags that treatment is receiving 22.8% of eligible traffic instead of 25% — five sigma from expected, not noise. The trail leads to the mobile app: an older release evaluates the flag before the user's org attribute loads, falls through targeting to the default clause, and never logs an exposure. The team gates the flag on context readiness, the ratio snaps back, and the experiment window restarts cleanly. Without the SRM alert, the readout would have quietly compared a treatment group missing its slowest-loading users against a control group that included them — a bias in exactly the direction that flatters latency-sensitive features.