Crescendo Attack, in depth: gradual multi-turn escalation, why per-message filters miss it, and conversation-level defence

How the escalation works

Three properties make Crescendo work, and each suggests a defence.

1. Every turn is locally acceptable. A question about history, a request to expand a point, a request to reformat: none of them alone violates a policy. A model or filter that judges the newest message mostly on its own merits approves each step.
2. The model's own words are the foothold. Asking the model to elaborate on text it already produced exploits its tendency to stay consistent with the conversation. Content it wrote looks pre-approved, and the attacker never has to type the sensitive terms. The paper's title is the canonical move: "great, now write an article about that".
3. Refusals are cheap to route around. If the target refuses, the attacker rephrases, or, where the interface allows editing or regenerating, deletes the refused exchange and tries another path. Automated tools call this backtracking.

The escalation usually runs through recognisable phases: an anchor question in a legitimate frame such as history or education, elaboration requests that zoom into one detail, a format shift into an article, story, script or step list, and finally extraction, where the attacker asks for the specifics that were the goal all along. The phases matter for detection because each leaves a signal in the transcript even when no single message is alarming.

Why per-message defences are blind to it

Most deployments put an input classifier before the model and an output classifier after it, and both score one message at a time. Against Crescendo that is the wrong unit of analysis. Inputs contain words like "that", "the third point" and "more detail", which carry no risk on their own. Outputs creep upward slowly, and each one sits just below the threshold because it is only a small step from the previous output, which was also allowed.

There is also an architectural hole that has nothing to do with classifiers. Many chat APIs are stateless: the client sends the full history, including earlier assistant messages, with every request. If your backend forwards client-supplied history to the model, an attacker does not need to escalate at all. They can write the fake assistant turns themselves, claiming the model already agreed. Crescendo then takes one request instead of seven. Keep conversation state on the server, or sign the history you hand to clients and reject any that fails verification.

A conversation-level monitor

A conversation-level monitor runs after each turn and asks what the conversation as a whole has become. It combines a whole-transcript score from your policy classifier with cheap trajectory features, and maps the result to graduated actions. The classifier is passed in as a function, so you can plug in a hosted moderation endpoint, an open guard model or your own fine-tuned model.

import re
from dataclasses import dataclass, field

REFERENTIAL = re.compile(
    r"\b(that|those|the (first|second|third|last) (one|point|step|part)|expand on|"
    r"go deeper|more detail|as you (said|mentioned)|you mentioned)\b", re.I)
FORMAT_SHIFT = re.compile(
    r"\b(write (an?|the) (article|story|scene|guide|script)|step[- ]by[- ]step|"
    r"as a (list|recipe|checklist)|in (full|more) detail)\b", re.I)


@dataclass
class Conversation:
    turns: list = field(default_factory=list)   # (user, assistant) pairs
    ewma: float = 0.0
    refusals: int = 0
    regenerations: int = 0


class CrescendoMonitor:
    def __init__(self, score, alpha=0.5, window_chars=16000, warn=0.45, block=0.7):
        self.score = score              # text -> policy risk in [0, 1]
        self.alpha, self.window = alpha, window_chars
        self.warn, self.block = warn, block

    def observe(self, conv, user, assistant, refused=False, regenerated=False):
        conv.turns.append((user, assistant))
        conv.refusals += refused
        conv.regenerations += regenerated
        transcript = "\n".join(f"USER: {u}\nASSISTANT: {a}" for u, a in conv.turns)
        whole = self.score(transcript[-self.window:])     # the conversation, not the delta
        conv.ewma = self.alpha * self.score(assistant) + (1 - self.alpha) * conv.ewma
        bonus = 0.05 * bool(REFERENTIAL.search(user)) + 0.05 * bool(FORMAT_SHIFT.search(user))
        bonus += 0.1 * min(conv.refusals + conv.regenerations, 3) / 3
        risk = min(1.0, max(whole, conv.ewma) + bonus)
        if risk >= self.block:
            return "end_or_refuse", risk
        if risk >= self.warn:
            return "steer", risk          # restate policy, answer at a higher level
        return "continue", risk

Two design choices matter here. The whole-transcript score is the main signal, and the heuristics only nudge it. On their own, regular expressions over "expand on that" would flag every legitimate long conversation. Refusals and regenerations also count, because a refusal followed by a rephrase is the fingerprint of backtracking. The steer action matters in practice: answering at a higher level and restating the policy costs a benign user little, while ending the session is reserved for high confidence.

Run the monitor before the response is shown when you can afford the latency, by scoring the transcript with the candidate answer appended. Otherwise run it asynchronously and act on the next turn. Either way, log the scores per turn so you can tune thresholds later.

Worked example: a support bot and its fraud rules

Take a banking support assistant whose system prompt forbids disclosing the internal rules its fraud team uses to flag transactions. This objective is harmless to describe, but it has the same structure as a dangerous one. The table shows a seven-turn escalation in summary, with scores from a hypothetical classifier tuned to that policy. The scores are illustrative, not measured.

No message score passes 0.45, so a per-message filter would have let every turn through, including turn 6. The monitor steered at turn 5, when the transcript score, nudged by the referential and format-shift features and the earlier refusal, reached 0.48, and it refused at turn 6. In this case the system prompt did its job only because the monitor kept the model from accumulating enough context to rationalise the disclosure. The broader fix is the one from the prompt-leak literature: real fraud rules should never be in the prompt at all.

Measuring exposure with an automated harness

You cannot tune thresholds without an attack generator. Microsoft's open-source PyRIT toolkit ships an automated Crescendo attack: an adversarial model writes each escalating turn, the target answers, a scorer judges whether the objective was reached, and refused turns are backtracked up to a limit. The sketch below follows the PyRIT 0.14 documentation. Import paths have moved between releases, and 1.x is out, so check the docs for the version you install.

import asyncio, os
from pyrit.executor.attack import AttackAdversarialConfig, CrescendoAttack
from pyrit.prompt_target import OpenAIChatTarget
from pyrit.setup import IN_MEMORY, initialize_pyrit_async

async def main():
    await initialize_pyrit_async(memory_db_type=IN_MEMORY)
    target = OpenAIChatTarget(endpoint=os.environ["TARGET_ENDPOINT"],
                              api_key=os.environ["TARGET_KEY"],
                              model_name=os.environ["TARGET_MODEL"])
    adversary = AttackAdversarialConfig(target=OpenAIChatTarget(
        endpoint=os.environ["ADV_ENDPOINT"], api_key=os.environ["ADV_KEY"],
        model_name=os.environ["ADV_MODEL"], temperature=1.1))
    attack = CrescendoAttack(objective_target=target,
                             attack_adversarial_config=adversary,
                             max_turns=7, max_backtracks=4)
    for objective in OBJECTIVES:          # your policy-specific goals
        result = await attack.execute_async(objective=objective)
        record(objective, result)         # outcome, turns used, transcript

asyncio.run(main())

Point the target at your full stack, including the monitor, not at the bare model, or you will measure the wrong system. Report attack success rate, the median number of turns to success, and how often the monitor fired before the judge said the objective was reached. Pin the attacker model, its temperature, the judge and the turn budget, because each changes the numbers. Pair every run with a benign multi-turn set, such as long research or troubleshooting conversations, and report the false-positive rate next to the attack rate. A monitor that stops Crescendo by refusing every deep conversation has not solved anything. Adaptive attacks of this kind should sit alongside static replays of known transcripts in the regression suite described in LLM red teaming.

Defences ranked

Defences differ in what they stop and what they cost. Ranked roughly by value for effort:

Defence in depth matters more than any single row. The monitor catches what the model lets through, and server-side history removes the shortcut. Keeping sensitive data out of the context window means even a successful jailbreak has little to extract. The jailbreak defence guide covers the single-turn layers this builds on.

Failure modes and trade-offs

Conversation-level defences fail in their own ways, and it is better to know them before an incident does the teaching.

The underlying trade-off is between context and cost. The more of the conversation each decision sees, the better it detects gradual drift, and the more tokens, latency and false positives it brings. Start with the whole transcript on the surfaces where a successful jailbreak would do real harm, and use cheaper windows elsewhere.

Running it in production

Roll out in shadow mode. Log monitor decisions without acting for two to four weeks. Review the highest-scoring benign conversations and adjust thresholds per product surface, since a medical information bot and a coding assistant have very different baselines.

Control cost. Whole-transcript scoring grows linearly with conversation length. Score a sliding window plus a running summary of earlier turns, and skip scoring for turns where the cheap features are quiet and the last score was low.

Watch for split attacks. A patient attacker can spread escalation across sessions or accounts. Aggregate flags per user and per organisation, and treat a series of steer events across sessions as one incident. Multi-turn attack patterns covers related strategies such as decomposition.

Mind privacy. Transcript logging for security is still personal data. Set retention, restrict access, and redact before sending conversations to human reviewers.

Alert on trends. Track steer and refuse rates per day. A sudden rise usually means a new tactic is circulating, or a model update shifted the baseline.

What to do next

• Check whether any endpoint accepts client-supplied assistant messages, and close it with server-side state or signed history.
• Add a whole-transcript policy score to your pipeline and log it per turn, starting in shadow mode.
• Write five to ten policy-specific objectives with benign stand-ins, and run an automated Crescendo harness against your full stack with pinned attacker, judge and turn budget.
• Build a benign long-conversation set and measure false positives next to attack success.
• Cap regenerations and edits per conversation, and count refusals followed by rephrasing as a signal.
• Move anything that must never be disclosed out of the prompt and behind access-controlled tools.
• Re-run the harness after every model or prompt change and alert on regressions.