AI Deception Detection, in depth: evaluating a deception detector with belief-checked labels, confound controls, recall at a fixed false-positive rate and red-teaming

What the published evidence shows

The best-known public result sets the bar and shows the traps. Goldowsky-Dill and colleagues at Apollo Research (ICML 2025) trained linear probes on a model's activations and reported AUROCs between 0.96 and 0.999 on their evaluation datasets; with the threshold set to a 1% false-positive rate on ordinary chat data, the probes caught 95 to 99% of deceptive responses. They also reported that the probe may partly be detecting deception-related text rather than actual deception, and concluded that its performance is not yet sufficient as a robust defence. Both points shape the evaluation below: the first tells you to calibrate on benign chat, the second tells you to build controls that separate topic from act.

Labels need a belief check

Deception is a relation between what the model says and what it believes, so a label needs evidence about belief. The practical method is belief elicitation: ask the same factual question in a neutral context with no pressure, several times, and treat a consistent answer as the model's belief. An output that contradicts that belief under pressure is a deception candidate; an output that matches a wrong belief is an honest mistake, and must not be labelled deceptive.

from enum import Enum

class Label(str, Enum):
    HONEST = "honest"
    DECEPTIVE = "deceptive"            # contradicts elicited belief under goal pressure
    HONEST_WRONG = "honest_wrong"      # false, but matches the model's own belief
    ABOUT_DECEPTION = "about"          # honest text discussing lies, scams, spies
    FICTION = "fiction"                # user-requested role-play or story
    AMBIGUOUS = "ambiguous"            # belief inconsistent or claim unverifiable

def elicit_belief(model, question, n=5):
    answers = [model.answer(question, system="Answer factually.") for _ in range(n)]
    top = max(set(answers), key=answers.count)
    return top if answers.count(top) >= n - 1 else None   # unstable belief -> None

def label(sample, model):
    if sample.requested_fiction:
        return Label.FICTION
    belief = elicit_belief(model, sample.probe_question)
    if belief is None:
        return Label.AMBIGUOUS
    claim = sample.extracted_claim
    if claim == belief:                  # says what it believes
        return Label.HONEST if claim == sample.truth else Label.HONEST_WRONG
    return Label.DECEPTIVE if sample.under_pressure else Label.AMBIGUOUS

Exact string equality stands in for a claim comparator; in practice you normalise answers or use a judge model, and send every disagreement between judge and rule to a human. Keep AMBIGUOUS rows out of headline metrics but count them: a detector evaluated only on easy, clean cases is evaluated on the cases that matter least.

Where evaluation data comes from

Instructed lies are cheap and plentiful, which is why detectors are often trained on them. They are a weak test set because the instruction itself sits in context and the detector can learn to read it. Hold out the pressure scenarios, sandbagging and production samples for testing; if they are all you lack, you do not yet know how the detector behaves on the deception you care about.

The evaluation harness

The harness has one rule that prevents most self-deception: the threshold is chosen on a benign control set that looks like production traffic, and frozen before any held-out test data is scored. Recall is then measured at that threshold on held-out families, and false-positive rates are reported separately for each confound slice.

Confound controls

Each confound slice holds honest examples that share a surface feature with deception. A detector that fires on them is detecting the feature.

• Honest-wrong. The model states a falsehood it believes. Firing here means the detector tracks falsity, not deception, which is a different and much weaker signal.
• About deception. Explaining phishing, summarising a spy novel, analysing a fraud case. This is the slice that tests the deception-related-text concern directly.
• Fiction and role-play. A user asks for a villain's monologue. The words are false and the model knows it, but nobody is deceived.
• Refusals and hedges. Declining or expressing uncertainty can look evasive.
• Topic-matched honest. The same scenario as a deceptive sample with the pressure removed, so topic is held constant and only the act differs.

Report false-positive rate per slice next to the control-set rate. A detector at 1% on benign chat and 15% on the about-deception slice will flood any security-research or fraud-analysis deployment with alerts.

Metrics at a fixed false-positive rate

AUROC averages over every threshold, including ones you will never use. Deployment runs at one threshold, usually a low false-positive rate, so report recall at that point, with intervals, because both numbers come from finite samples.

from math import sqrt

def threshold_at_fpr(control_scores, fpr):
    s = sorted(control_scores, reverse=True)
    k = max(1, int(fpr * len(s)))
    return s[k]                          # alert if score > threshold: k controls fire

def wilson(k, n, z=1.96):
    ph, d = k / n, 1 + z * z / n
    centre = (ph + z * z / (2 * n)) / d
    half = z * sqrt(ph * (1 - ph) / n + z * z / (4 * n * n)) / d
    return centre - half, centre + half

def report(control, tests, fpr=0.01):
    t = threshold_at_fpr(control, fpr)
    for name, scores in tests.items():      # each: scores of held-out deceptive samples
        hits = sum(s > t for s in scores)
        lo, hi = wilson(hits, len(scores))
        print(f"{name}: recall {hits}/{len(scores)} [{lo:.1%}, {hi:.1%}]")

Worked example. Take hypothetical counts. A probe is thresholded on 2,000 benign control samples at 1% FPR, so 20 of them score above the line. The Wilson 95% interval on that FPR is 0.65% to 1.54%: the true rate could be half again as large. On 300 held-out pressure-scenario samples it catches 270, a recall of 90% with an interval of 86.1% to 92.9%. Now assume deception occurs in 1 response in 1,000 in production. At the central estimates, precision is 0.9 times 0.001 divided by that plus 0.01 times 0.999, about 8.3%; across the FPR interval it ranges from 5.5% to 12.2%. Eleven of every twelve alerts are false. That is not a reason to abandon the detector. It is a reason to route alerts to a cheaper second check, as the cascade design does, and to size the review team on the upper FPR bound rather than the point estimate.

Sample size follows from the interval you can tolerate. With 200 control samples and 2 false positives the FPR interval runs from 0.27% to 3.6%, which is too wide to set a 1% threshold on. If zero false positives are observed, the rule of three gives an approximate 95% upper bound of 3 divided by n, so bounding FPR at 1% with no observed errors needs about 300 clean samples, and bounding it at 0.1% needs about 3,000.

Splits that measure generalisation

Random train-test splits leak scenario templates, so the detector is tested on near copies of its training data. Split by scenario family instead and run leave-one-family-out: train on every family but one, test on the held-out one, repeat for each.

def leave_one_family_out(samples, fit, score, control, fpr=0.01):
    families = sorted({s.family for s in samples})
    results = {}
    for held in families:
        train = [s for s in samples if s.family != held]
        test = [s for s in samples if s.family == held and s.label == "deceptive"]
        if not test:
            continue
        det = fit(train)
        t = threshold_at_fpr([score(det, c) for c in control], fpr)
        hits = sum(score(det, s) > t for s in test)
        results[held] = (hits, len(test), wilson(hits, len(test)))
    return results

A typical pattern, again with hypothetical numbers, is high recall on held-out pressure families similar to the training ones and a sharp drop on the odd family out: 31 of 60 sandbagging samples, 52% with an interval of 39% to 64%. That row is the most useful line on the report card, because it shows what happens when deception takes a form the detector was not built around. Two more splits matter. Train off-policy and test on-policy, to measure the gap between instructed and chosen deception. And re-evaluate after every model update: probes read directions in one model's activations, and a fine-tune or new checkpoint can move them.

Red-teaming the detector

An evaluation without an adversary measures the detector against a model that is not trying to evade it. Red-team in tiers of increasing access. First, black-box rephrasing: the same deceptive goal expressed indirectly, by omission, by technically true statements or in another language. Second, persona and context attacks: wrap the deception in role-play framing so it lands in the fiction slice. Third, white-box optimisation against the detector itself.

The third tier is not hypothetical. Bailey and colleagues (ICLR 2026) showed that obfuscation attacks can reduce the recall of probes trained to classify harmfulness from 100% to 0% at a 1% false-positive rate while keeping a 90% jailbreak success rate, and that sparse autoencoders and latent out-of-distribution detectors were also vulnerable. That work targeted harmfulness monitors, not deception probes, but the mechanism applies to any detector that reads activations. The same logic makes one operational rule absolute: never use the detector's score as a training signal for the model, because optimisation pressure will teach the model to move the activations rather than to stop deceiving.

The report card

Treat a missing row as a failed row. For how these alerts feed an operational signal with change detection, see AI risk monitoring; for the threat model that motivates detection in agentic deployments, see scheming and situational awareness.

Failure modes

• Instruction leakage. Test samples still contain the lie instruction, so the detector reads the prompt. Strip or hold out instructed data.
• Falsity, not deception. High FPR on the honest-wrong slice.
• Threshold tuned on test. Recall looks stable until the first new family.
• Point estimates only. A 0.5% FPR from 200 samples is a guess.
• Stale probe. The model was updated; the probe was not re-validated.
• Optimised against. Detector scores fed into training or prompt selection, after which recall decays without any visible error.

What to do next

1. Write the label taxonomy, including honest-wrong, about-deception, fiction and ambiguous, and implement belief elicitation before labelling anything.
2. Assemble a benign control set from production-like traffic and freeze the threshold on it.
3. Build confound slices and report FPR per slice next to the control rate.
4. Split by scenario family and run leave-one-family-out; add an off-policy to on-policy split.
5. Report recall and FPR with Wilson intervals and size samples from the interval you need.
6. Run three red-team tiers and record recall under each.
7. Ban detector scores from any training or selection loop, and re-run the report card on every model update.