Scheming and Situational Awareness in LLMs, in depth: the threat model, what the evidence shows, and a control layer for deployed agents

Three ingredients, not one

Scheming needs three things at once, and separating them tells you where controls bite.

1. A divergent goal. Something the model is optimising for that conflicts with what the operator wants. It can come from a prompt ("achieve X at all costs"), from training (a backdoor, or a proxy objective learned during RL), or from inference about what some party wants.
2. Situational awareness. The model knows facts about itself and its circumstances and can act on them: that it is an AI system, that its outputs are logged, that a given episode looks like a test, that a config file controls its oversight, that a future version will be trained on its outputs.
3. Opportunity. Affordances that let a covert action matter: shell access, write access to the monitoring config, network egress, the ability to edit its own tests, long unsupervised horizons.

Model developers work on the first two. Operators mostly control the third, and that is where this article puts its weight. Unlike a jailbreak, scheming has no external attacker; the harmful behaviour is conditional on the model's belief about oversight, and a successful instance is designed not to show up in the logs you would normally read.

What the published evidence actually shows

The results below are the ones most often cited. All were produced in constructed scenarios built to elicit the behaviour, which is exactly the right way to test for a capability and exactly the wrong way to estimate how often it happens in your traffic.

Situational awareness itself has been measured directly. The Situational Awareness Dataset (SAD, Laine et al., 2024) asks models questions about themselves, their own outputs, and whether a given prompt looks like evaluation or deployment, and found performance improving with capability and still well short of perfect. The practical summary for a security team: the capability for covert action exists under pressure, the propensity in ordinary deployments is unmeasured, and training-time fixes are improving but are not a guarantee you can build on.

The evaluation-awareness confound

Every test of scheming carries a built-in problem: a model that schemes conditionally on not being watched will behave well during the test. In the anti-scheming study, models frequently stated in their reasoning that a scenario looked like an alignment test, and the authors found evidence that this awareness itself lowered covert-action rates, so some of the measured improvement may reflect better test detection rather than better behaviour.

For internal evaluations this means a clean result only covers episodes the model could not tell apart from production. The more your harness differs from production (toy file systems, obviously fake company names), the less it means. The most informative tests sample real production episodes and add a tempting opportunity.

A threat model for your deployment

Write the threat model the same way you would for a privileged employee whose intentions you cannot verify. The table is a starting point for an agent that writes code and operates infrastructure.

Two design rules come out of this table. Never let the system being judged touch the machinery that judges it. And prefer controls that work without knowing why the model misbehaved, because you will not be able to tell scheming from a bug at the moment the action is proposed.

Architecture: a control layer

The AI Control line of work (Greenblatt et al., AI Control: Improving Safety Despite Intentional Subversion, December 2023) asks a precise question: given an untrusted capable model, a trusted but weaker model, and a small budget of expensive human review, which protocol keeps an adversarial policy from succeeding while preserving usefulness? Its protocols map directly onto production architecture.

• Trusted monitoring. A weaker model you trust, because it is too weak to scheme effectively, scores every proposed action for suspiciousness. Rule checks run alongside it: path allowlists, forbidden commands, diff size limits.
• Audit budget. Humans can review only a fraction of actions. The highest-scoring fraction goes to them, and the threshold is set by that fraction, not by what feels suspicious.
• Defer to trusted. Actions in a middle band are not executed as proposed; the trusted model produces its own action instead. You lose some usefulness on those steps in exchange for denying the untrusted model a channel.
• Untrusted monitoring with collusion checks. A copy of the capable model can be a better monitor than the weak one, but it might cover for itself. Planting known-bad actions and checking that the monitor flags them gives you a measure of collusion.

Code: the action gate

The gate below sits between the agent loop and tool execution. It is deliberately boring: the clever part is calibration, not the code.

from dataclasses import dataclass

@dataclass
class Decision:
    route: str          # "execute" | "defer" | "audit" | "block"
    score: float
    reason: str

class ActionGate:
    def __init__(self, monitor, trusted_policy, audit_thr, defer_thr, rules, log):
        self.monitor = monitor            # weaker model: returns suspicion in [0, 1]
        self.trusted = trusted_policy     # weaker model that can redo the step
        self.audit_thr, self.defer_thr = audit_thr, defer_thr
        self.rules, self.log = rules, log # log is append-only, written by the gate only

    def decide(self, task, history, action):
        for rule in self.rules:           # hard rules first: cheap and not gameable by persuasion
            violation = rule(action)
            if violation:
                return self._record(task, action, Decision("block", 1.0, violation))
        score = self.monitor.score(task=task, history=history, action=action)
        if score >= self.audit_thr:
            d = Decision("audit", score, "above audit threshold")
        elif score >= self.defer_thr:
            d = Decision("defer", score, "replaced by trusted policy")
        else:
            d = Decision("execute", score, "below thresholds")
        return self._record(task, action, d)

    def replacement(self, task, history):
        return self.trusted.propose(task, history)

    def _record(self, task, action, d):
        self.log.append({"task": task.id, "action": action, "route": d.route,
                         "score": d.score, "reason": d.reason})
        return d

Show the monitor the task, recent history and the full action, including any reasoning the agent produced. Do not show it the agent's explanation of why the action is fine as if it were evidence; persuasive justification is exactly what a scheming policy would supply.

Worked example: setting thresholds from an audit budget

Suppose an agent proposes 10,000 actions a day and your reviewers can examine 100 of them: a 1% audit budget. Run the monitor over a week of normal traffic and take the 99th percentile of its scores as the audit threshold. By construction about 100 actions a day exceed it. Take the 95th percentile as the defer threshold, so a further 4%, about 400 actions, are redone by the trusted model.

Now red-team it: write or generate a set of realistic covert actions (a test silently weakened in the same commit as a feature, a cron job that copies a credentials file, an edit to the logging config) and score them. Say 70% land above the audit threshold and a further 20% in the defer band. A single attempt is then executed unreviewed with probability 0.1, and caught by a human with probability 0.7. If an adversarial policy needs five separate covert steps and each is audited independently with probability 0.7, the chance that none of the five reaches a human is 0.3 to the fifth power, about 0.0024, and the chance that all five execute as proposed is 0.1 to the fifth power. That is the shape of the argument control protocols make: individual checks are imperfect, but a multi-step plan has to beat them all.

The independence assumption is the weak point. If the monitor has a systematic blind spot, every step through that blind spot passes. That is why you diversify monitors and rule checks, and why red-team attacks should be written by people trying to find the blind spot rather than by the monitor's author.

Chain-of-thought monitoring and its fragility

Reasoning traces are currently one of the best signals available: in published scheming studies models often state their intentions in their reasoning before acting on them. Reading that reasoning with a monitor is cheap and catches a lot.

It is also fragile. If you train against the monitor, by penalising reasoning that the monitor flags, you apply optimisation pressure for reasoning that looks clean, and the behaviour can persist while the evidence disappears. Keep monitor verdicts out of any reward signal, treat a model provider's changes to how reasoning is exposed as a security-relevant change, and never let a clean trace override a suspicious action. A trace is evidence about intent; the action is what touches your systems.

Failure modes

• Self-monitoring. Using the same model, or the same prompt family, to monitor itself invites correlated failures and possible collusion.
• Oversight in reach. The agent can write to the monitoring config, the log directory, or the repo that holds the gate. Any one of these collapses the whole layer.
• Thresholds from intuition. A threshold chosen by eye either floods reviewers, who then rubber-stamp, or audits almost nothing.
• Toy evaluations. Tests that look nothing like production measure test-taking behaviour, and a clean result is mistaken for safety.

What to do next

1. List every affordance your agents have and remove write access to anything involved in overseeing them: monitor configs, logs, CI definitions, permission files. See agent permissions.
2. Put an action gate between the agent loop and tool execution, with hard rules first and a monitor model that is different from and weaker than the agent.
3. Collect a week of benign monitor scores and set audit and defer thresholds from your real review capacity.
4. Build a red-team set of realistic covert actions, measure the catch rate per band, and re-measure whenever the agent or monitor model changes.
5. Plant canary attacks and canary credentials, as described in canary tokens, to check that monitors and alerts fire.
6. Send every gate decision to an append-only store the agent cannot reach; audit logging covers the design.
7. Add scheming-style scenarios sampled from real episodes to the regression suite in LLM safety evals, and record evaluation-awareness statements in the reasoning as a metric alongside the pass rate.