Why architecture matters here
Architecture matters here because a bandit is a feedback loop wired directly into live traffic, and feedback loops are exactly the systems where a small structural mistake compounds into a large, silent failure. If rewards are attributed to the wrong arm, if the delayed-reward join is subtly broken, or if the exploration rate is mis-set, the bandit will confidently converge on the wrong variant and keep serving it, and because it looks like it is working — traffic is flowing, an arm is winning — nobody notices until a downstream metric sags. The architecture has to make attribution correct and the exploration honest, or the whole loop misleads.
The core value the bandit delivers is reduced regret. Regret is the cumulative reward you gave up by not always playing the best arm; an A/B test's regret is large because it deliberately serves the losing arm to half of traffic for the whole test. A good bandit drives regret down over time by concentrating traffic on winners as it becomes confident, while still spending a little on the others to stay correct. On a high-volume service, the difference between a bandit and a fixed test is measured in real outcomes — resolved tickets, conversions, revenue — not just statistical elegance.
The second structural reason is adaptivity to change. Models are retrained, prompts are edited, a new distilled variant is added to save cost. In an A/B-test world, each change starts a new fixed experiment; in a bandit world, a changed or added arm simply re-enters the exploration budget and the loop re-balances. This is especially valuable when the environment itself is non-stationary — user behavior shifts, an upstream model changes — because a bandit with a memory that decays old evidence tracks the moving optimum instead of being anchored to a stale test result.
But adaptivity is also where the danger lives, and the architecture has to contain it. A bandit that shifts allocation too aggressively on thin evidence will starve a good arm before it has had a chance to prove itself, permanently. That is why the guards in the design — a minimum traffic floor per arm, a reward floor below which an arm is pulled entirely — are not optional decorations; they keep the exploitation from eating the exploration and turning a learning loop into a premature commitment.
Finally, the delayed-reward reality shapes everything. In most real systems the reward does not arrive with the request: a click comes seconds later, a conversion hours later, a resolved ticket a day later. The architecture must hold the pull, wait for the reward, and join the two correctly — and it must reason about arms whose rewards have not landed yet, so it does not treat an un-joined pull as a failure. Getting this delayed-attribution machinery right is most of the engineering, and it is what separates a bandit that learns the truth from one that learns an artifact of its own timing.
The architecture: every piece explained
The top row is the serving path. A request arrives carrying whatever context is available and the set of eligible arms. The policy chooses an arm — this is the bandit algorithm proper. Epsilon-greedy plays the current best arm most of the time and a random arm a small epsilon fraction of the time. UCB (upper confidence bound) picks the arm with the highest optimistic estimate, adding a bonus that shrinks as an arm is pulled more, so under-explored arms get a look. Thompson sampling maintains a posterior distribution over each arm's reward and samples from it, so an arm the bandit is uncertain about is chosen in proportion to the probability it is best. The chosen variant is then served.
The bottom-right of the loop is reward collection. Sometime after serving, the system observes a reward — the user clicked, the ticket resolved, the conversion fired — and that reward must be attributed back to the arm that was pulled for that specific request. The update arm stats step folds the reward into that arm's running estimate: its mean reward and, importantly, its uncertainty. Because the estimate moved, the policy's future choices shift, which is the arrow labeled 'shift allocation' — this is the learning closing the loop.
The delayed reward buffer on the left is the piece naive implementations forget. Between pulling an arm and seeing its reward there is a gap — sometimes long. The buffer holds the association from request to arm so that when the reward eventually arrives, possibly out of order and possibly much later, it can be joined to the correct arm. Without this buffer the bandit either loses rewards it cannot attribute or, worse, attributes them to whatever arm happens to be current, poisoning the statistics.
The exploration budget control tunes how much the bandit is willing to spend on learning versus earning. In epsilon-greedy it is the epsilon; in UCB it is the width of the confidence bonus; in Thompson sampling it is the prior width and how fast the posterior tightens. Set it too low and the bandit commits early on noise; set it too high and it wastes traffic exploring arms it already knows are worse. Often the budget is annealed — high at launch when nothing is known, lower as evidence accumulates.
The guardrails box is what keeps the bandit safe in production. A minimum traffic per arm floor guarantees every arm keeps getting a trickle of traffic so a temporarily-unlucky good arm is never permanently starved, and so you retain a live measurement of every option. A reward floor lets you pull an arm entirely if its measured reward drops below an acceptable threshold — a safety valve against serving a genuinely broken variant. The ops strip tracks the health signals: pulls per arm, each arm's reward mean and confidence interval, cumulative regret, how fast allocation is drifting, and the lag on the delayed-reward join.
End-to-end flow
Follow a request through a Thompson-sampling bandit choosing among three model variants for a support-answer service. The request arrives with its context and the three eligible arms. The policy draws one sample from each arm's posterior reward distribution — arm A is well-explored with a tight posterior around 0.62, arm B is newer with a wide posterior centered at 0.55, arm C is barely tried with a very wide posterior. On this particular draw, arm B's wide posterior happens to sample high, so the bandit pulls arm B and serves its answer. This is exploration driven by uncertainty: B got chosen not because it is known best but because it might be, and the bandit wants to find out.
The answer is served, and the request-to-arm association — this request went to arm B — is written into the delayed reward buffer. Nothing more happens immediately, because the reward for a support answer is not instantaneous: it depends on whether the user marks the answer helpful or comes back with a follow-up, which unfolds over the next minutes. The bandit does not wait; it goes on serving other requests, each routed by a fresh draw from the posteriors.
A few minutes later the reward lands: the user marked arm B's answer helpful, a reward of 1. The system looks up the buffer, finds that this outcome belongs to the pull of arm B, and runs the update: arm B's posterior shifts up and tightens slightly, because it now has one more positive observation. Had the reward been negative, the posterior would have shifted down. Either way the update is attributed to B specifically — not to whatever arm is current now — which is exactly why the buffer matters.
Over thousands of such requests the posteriors sharpen. Suppose arm B's true reward really is higher than A's; its posterior climbs and narrows, and Thompson sampling starts drawing B as the winner more and more often, so allocation drifts toward B — exploitation. Arm C, meanwhile, has been getting occasional exploratory pulls thanks to its wide posterior; if those come back poor, its posterior settles low and it is chosen rarely, though the minimum-traffic floor ensures it never drops to exactly zero, so if C silently improves later the bandit will notice. The allocation is now mostly B, some A, a little C — and it got there without a fixed test ever being declared.
Step back and see the properties the loop produced. Regret stayed low because traffic concentrated on the best arm as soon as the evidence justified it, not after a fixed test window. Exploration was automatic and proportional to uncertainty, so the bandit spent the least traffic on options it was already confident about. Rewards were attributed correctly despite arriving late and out of order, because the buffer held the association. And every arm stayed measurable thanks to the traffic floor, so the system remained responsive to change. All of it is a direct consequence of the loop: choose by policy, buffer the pull, join the delayed reward, update the arm, let allocation follow.