Human Approval Gates That Keep Agents Safe Without Becoming the Bottleneck

What a gate really catches

Start with the probability, because it explains almost every design mistake. Suppose a fraction h of an agent's tool calls would cause harm if executed: a refund to the wrong account, an email with someone else's data, a delete with a bad filter. The gate sends a fraction of calls to a human. A harmful call is stopped only if it is both routed to a person and recognised by that person. The residual harm per call is:

harm_executed = h * [ P(not gated | harmful) + P(gated | harmful) * (1 - d) ]

h  = share of calls that are harmful
d  = probability a reviewer recognises a harmful call they are shown

Two quantities matter, and they pull against each other. Coverage is P(gated | harmful): how well the routing rule sends dangerous calls to people. Detection d is how good those people are at the moment they look. Gating everything maximises coverage at 100%, but detection is not a constant. It depends on how many decisions a reviewer is making, how much context they get, and how often the right answer has been "approve" recently. A gate with 100% coverage and 70% detection lets through 30% of harmful calls. A gate with 90% coverage and 97% detection lets through 0.10 + 0.90 × 0.03 ≈ 12.7%. The narrower gate is more than twice as safe.

This is the core of the topic: an approval gate is a scarce attention budget, and the design problem is where to spend it.

Reviewer fatigue and the rubber stamp

Human factors research on automation has a name for what happens when people supervise a system that is usually right: complacency, or automation bias. When nearly every proposed action is fine, the cost of looking carefully feels high and the reward feels low, so people start confirming rather than checking. Vigilance research adds that sustained monitoring of rare events degrades over time. Neither gives you a formula for your reviewers; measure it, and design as if it exists.

The lab uses an explicitly made-up curve so the effect is visible: detection is 97% while a reviewer has made at most 30 decisions in the last hour, then falls linearly to 47% at 90 decisions per hour.

Three production signals tell you the rubber stamp has arrived:

• Median time to decision falls while queue length rises. People are clearing the queue, not reviewing it.
• Denial rate drifts toward zero. If your agent produces bad proposals at a steady rate and reviewers deny fewer of them every week, detection is falling.
• Canary calls pass. Seed the queue with a small number of synthetic, clearly wrong proposals (a refund to a test account that does not exist, an email to an obviously wrong recipient) and track how many are approved. This is the only direct estimate of d you will get, and it is cheap.

Mission one in the lab starts from the most common setup: every write needs approval, two reviewers are on shift, and timed-out approvals go through automatically. The busiest reviewer makes about 64 decisions an hour, more than a third of harmful calls slip through, and the p95 wait hits the 15-minute timeout. The fix is not more diligence. It is sending fewer calls to people.

Queueing arithmetic: when the gate becomes the bottleneck

Approval is a service with a queue in front of it, so the standard utilisation check applies. If gated calls arrive at λ per minute, a reviewer takes s minutes per decision on average, and c reviewers are on shift, utilisation is ρ = λ·s / c. When ρ approaches 1 the queue and the wait grow sharply; above 1 they grow without limit until the shift ends or the timeouts start firing.

The table assumes 1.2 minutes per decision, which is what the lab uses. Utilisation alone underestimates the problem for two reasons. Arrivals are bursty, so a shift with 70% average utilisation still has hours above 100%. And fatigue couples the two failure modes: a busy reviewer is both slower to reach the next call and worse at judging it. Size the reviewer pool from your peak hour, not the daily average, and keep planned utilisation well under 1 so that a surge does not convert directly into either timeouts or rubber-stamping.

The decision time s is also yours to control. A card that shows the exact arguments, a readable diff of what will change, the agent's reason and the triggering context can be judged in seconds; a raw JSON blob cannot.

Choosing what to gate: risk scores and their blind spot

The obvious way to narrow the gate is to score each proposed call and send only high scores to a person. The score can come from rules, a classifier or a second model acting as a judge. A score gate is a real improvement: in the lab, moving from gate-every-write to a score threshold of 55 cuts the gated share from 55% to about 8%, brings reviewer load down to around 20 decisions an hour, and lowers harm because detection recovers.

The blind spot is that a score learns what harmful calls usually look like. A prompt-injection attack is designed so the resulting call looks ordinary. Indirect prompt injection, where instructions arrive inside content the agent reads (a web page, a product review, an email, a tool result), has been demonstrated against real LLM-integrated applications, and benchmarks such as AgentDojo exist precisely because agents with tools follow injected instructions at meaningful rates. An injected refund of a plausible amount to a plausible-looking account, triggered by text in a customer message, can score as low as a routine one.

In the lab, the injection event adds calls that are harmful but scored low (mean 30 on a 0 to 100 scale, standard deviation 10). With a threshold of 55, the share of injected calls that reach a human is the probability that a normal variable with mean 30 and standard deviation 10 exceeds 55: about 0.6%. The gate is effectively absent for the attack. Mission two asks you to survive this campaign, and the answer is to change what the gate keys on.

Tiering by reversibility, not by appearance

A more robust routing rule asks what the tool can do, not how this particular call looks. Sort tools into tiers by consequence and reversibility:

The last row matters as much as the others. The cheapest gate is the tool you never give the agent, which is also the main advice in the OWASP entry on excessive agency: limit the functions, permissions and autonomy an LLM-driven system has to what the task needs.

For reversible writes, an undo window replaces pre-approval with post-hoc review: the action runs, a reviewer or an automated check samples recent actions, and anything wrong is rolled back. The lab assumes 70% of harmful reversible calls that bypass the gate are caught and undone this way; your real figure depends on how fast and how thoroughly you audit. Reversible mistakes cost a correction; irreversible ones cost an incident.

In the lab the tiered policy sends about 30% of calls to people at 240 calls per hour: every refund, email and delete, plus reversible calls above the score threshold. That is more than the score gate, so it needs three reviewers instead of two, but injected refunds and emails now always reach a person, and harm falls back under 1.5 per 1,000 calls.

Timeouts: fail open, fail closed, or escalate

Every gate needs an answer to "nobody responded". There are three, and teams often pick the wrong one under delivery pressure.

• Fail open (auto-approve after the timeout) keeps throughput but turns the timeout into a bypass. The calls that time out are exactly the ones that arrived when reviewers were overloaded, which is when an attacker would choose to strike. If you would not run the call without approval, you should not run it because approval was slow.
• Fail closed (deny after the timeout) is safe for harm but fails legitimate work.
• Escalate or defer: return the task to a work queue with a clear status, notify the user, and let a person complete it later. This is fail closed with a better user experience and is usually the right default for irreversible actions.

Fail closed only works if capacity is planned for peaks. Mission three combines a refund surge (twice the call rate for two hours, with many more refunds) with half the reviewers away for overlapping hours, while timed-out approvals fail open. Switching to fail closed fixes harm immediately but drops legitimate completions below target, because the thin shift cannot clear the queue within 15 minutes. You have to decide together how long a customer can wait (the timeout), how many reviewers cover the peak, and which way the gate fails.

The knobs in real systems

Approval gates are increasingly first-class features rather than custom middleware. Check the exact API in the version you run; these areas are moving quickly.

• Model Context Protocol. The tools section of the specification says there should always be a human in the loop with the ability to deny tool invocations, and that clients should show which tools are exposed and prompt for confirmation on operations. Tools can carry annotations such as readOnlyHint, destructiveHint, idempotentHint and openWorldHint. These are hints from the server: the specification tells clients to treat them as untrusted unless the server is trusted, so use them to set a default tier, then override with your own policy.
• LangGraph. Human-in-the-loop is built on a checkpointer: a node can call interrupt() to pause the graph with a payload for the reviewer, and the graph resumes when you invoke it again with a resume command carrying the decision. Static breakpoints (interrupt_before and interrupt_after at compile time) also exist and are mainly meant for debugging. Persisted state lets a paused approval wait hours without holding a process open.
• Your own tool layer. Whatever the framework, the enforcement point should sit in the code that actually executes the tool, not in the prompt. A system prompt that says "ask before refunding" is a request to the model; a wrapper that refuses to call the refund API without a signed approval record is a control.

A minimal policy function, framework-agnostic, looks like this:

IRREVERSIBLE = {"issue_refund", "send_email", "delete_records"}
NEVER = {"change_payout_account", "grant_role"}

def route(call, risk_score, threshold=60):
    if call.tool in NEVER:
        return "reject"                       # not an agent capability
    if call.tool in IRREVERSIBLE:
        return "human"                        # always, whatever the score says
    if call.is_read_only:
        return "run"
    if risk_score >= threshold:
        return "human"
    return "run_with_undo"                    # reversible: audit and roll back

def on_timeout(call):
    return "defer_to_queue"                   # fail closed, tell the user, finish later

A worked example: sizing the gate for a support agent

Take a hypothetical Belgavi's Book Store support agent making 240 tool calls an hour across an eight-hour shift. Using the lab's synthetic mix, 45% are reads, 20% are internal notes, 8% are price changes, and 27% are refunds, emails or deletes. About 1.2% of calls would be harmful, roughly 12 per 1,000. All of these numbers are illustrative.

Gate every write. 55% of calls (132 an hour) go to people. At 1.2 minutes each that is 2.64 reviewer-hours of work per hour, so two reviewers cannot keep up; the queue grows, timeouts fire, and the busiest reviewer makes over 60 decisions an hour. Detection in the lab's curve falls to roughly 69%, and with timed-out approvals failing open the gate stops only around 60% of harmful calls.

Score gate at 55. Harmful calls in the lab have scores centred on 72 with a standard deviation of 13, so the chance one scores under 55 is about 9.5%. Gated volume drops to about 8%, reviewer load to about 20 decisions an hour, and detection returns to 97%. Expected leakage is 12 × (0.095 + 0.905 × 0.03) ≈ 1.5 per 1,000 calls, which matches what the lab reports. This is the best policy for ordinary mistakes.

Tiered with an injection campaign. Add a campaign that makes about 2.6% of all calls injected refunds or emails scored like routine traffic. The score gate misses almost all of them and harm jumps to over 25 per 1,000. The tiered policy gates every irreversible call (27% of traffic plus a few high-scoring reversible ones, about 30% in total), needs three reviewers to keep load near 35 decisions an hour, and brings harm back to about 1 per 1,000. The cost is one extra reviewer; the alternative is an attack that walks through a gate built for honest mistakes.

Failure modes and how to detect them

Most approval-gate incidents fall into a handful of patterns, and each has a metric you can alert on.

The most common mistake is that nobody owns the detection rate. Make one person responsible for the weekly numbers (share gated, peak reviewer load, p95 wait, timeout outcomes, canary pass rate, incidents by tier), and tier every new tool before the agent can call it.

The lab maps directly onto these failures. Mission one is rubber-stamping, mission two is score blindness, mission three is a timeout that fails open during a peak, and mission four doubles traffic under an active injection campaign and asks for low harm, high completion and a short wait with as few reviewers as possible.

Run the lab: Approval gates: stop harmful tool calls without drowning reviewers

The lab runs five synthetic eight-hour shifts of an agent's tool calls and draws the first one. Each dot is a call placed by time and risk score; the dashed line is the score gate. Blue calls were approved, green were harmful and stopped, red were harmful and executed, orange timed out, and teal were harmful reversible calls undone after the fact. The pink line underneath is the approval queue. Start with the default gate-every-write setup and watch the queue and reviewer load, then load mission one.

Prefer a full-width view? Open the standalone lab. Your stars are saved in this browser and shown on the series hub.

What to do next

1. List every tool your agent can call and assign each to a tier: read-only, reversible, irreversible or external, and never-by-agent. Remove the last group.
2. Move enforcement into the tool-execution layer so that gated tools refuse to run without a stored approval record.
3. Measure decisions per reviewer-hour at your peak hour and the p95 approval wait; plan reviewer capacity from the peak.
4. Seed a few clearly wrong canary proposals into the approval queue each week and track how many are approved.
5. Decide the timeout behaviour per tier explicitly; for irreversible tools, fail closed and defer to a work queue rather than auto-approving.
6. Redesign the approval card to show the exact change, the agent's reason and the triggering context so a decision takes seconds.

Sources and further reading

• Model Context Protocol specification: Tools (2025-06-18)
• LangGraph documentation: Human-in-the-loop
• OWASP Top 10 for LLM Applications 2025: LLM06 Excessive Agency
• Greshake et al., Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Debenedetti et al., AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
• Parasuraman and Manzey, Complacency and Bias in Human Use of Automation: An Attentional Integration (Human Factors, 2010)

Related in this series:

• Evaluating Agent Trajectories: Why pass@k Hides Unreliable Agents
• MCP Tool Poisoning: How One Malicious Description Hijacks an Agent
• Why Agent Loops Run Away and How Step Budgets Stop Them

Diagrams, examples and the lab model are original educational material. Lab numbers are calculated by a synthetic model, not measured on real hardware or services.

Views are my own and do not represent any employer, past or present.