Perplexity Filter Defense, in depth: scoring prompts for adversarial suffixes, windowed NLL, calibration and its limits

What perplexity measures, and why suffixes stand out

A causal language model assigns each token a probability given the tokens before it. For a sequence of N tokens, the average negative log-likelihood is the mean of -log p(x_i | x_<i) over the tokens, and perplexity is the exponential of that mean. The general metric is covered in perplexity and evaluation metrics. Two properties matter for defense.

First, ordinary prose scores low, because each word is fairly predictable from the context. A GCG suffix is chosen by gradient search to change the target model's behavior, with no pressure to stay fluent, so its tokens are individually improbable and its NLL is high.

Second, the mean dilutes. A 600-token benign request with a 20-token adversarial suffix has a mean dominated by the benign part. That is why Jain and colleagues, in "Baseline Defenses for Adversarial Attacks Against Aligned Language Models" (2023), proposed a windowed filter: slide a fixed window over the token NLLs and flag the prompt if any window's mean is too high. In their setup the window was 10 tokens. Alon and Kamfonas, in "Detecting Language Model Attacks with Perplexity" (2023), scored prompts with GPT-2, found many false positives from plain perplexity thresholds, and reduced them by training a LightGBM classifier on perplexity together with token length.

Architecture

The scorer does not need to be the target model, and usually should not be. A small open model such as GPT-2 (124M parameters) runs on CPU or a slice of a GPU in milliseconds for typical prompts. Using the target model would give a better-matched signal but doubles your inference cost and is impossible for hosted APIs that do not return prompt log-probabilities.

Choosing the scorer is a real decision. Its training data defines what counts as normal: GPT-2 saw mostly English web text, so it finds Hindi, Japanese and Python all surprising. If your traffic is multilingual or code-heavy, a small multilingual or code-trained model gives a far better separation between benign and attack text. Tokenizers matter too: a tokenizer that splits rare words into many byte-level pieces inflates per-token NLL on names and jargon. Whatever you pick, pin the exact model revision, because a silent model update moves every score and invalidates your threshold. Treat the scorer like any other dependency, with a version, a test set and an owner.

Implementation: per-token and windowed NLL

The common bug is to pass labels=input_ids to the model and read loss. That gives only the mean over the whole sequence, so no windowing is possible. Compute per-token NLL from the logits instead, and handle the scorer's context limit (1,024 tokens for GPT-2) by scoring in overlapping chunks:

import torch, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("gpt2")
lm = AutoModelForCausalLM.from_pretrained("gpt2").eval()

@torch.no_grad()
def token_nll(text: str, max_len: int = 1024, stride: int = 512) -> torch.Tensor:
    ids = tok(text, return_tensors="pt").input_ids[0]
    if len(ids) < 2:
        return torch.empty(0)
    out = torch.full((len(ids) - 1,), float("nan"))
    for start in range(0, len(ids) - 1, stride):
        chunk = ids[start:start + max_len]
        logits = lm(chunk.unsqueeze(0)).logits[0, :-1]
        nll = F.cross_entropy(logits, chunk[1:], reduction="none")
        # skip positions the previous chunk already scored; the rest get more context
        keep = 0 if start == 0 else max_len - stride - 1
        out[start + keep:start + len(nll)] = nll[keep:]
        if start + max_len >= len(ids):
            break
    return out

def features(text: str, window: int = 10) -> dict:
    nll = token_nll(text)
    n = len(nll)
    if n == 0:
        return {"n": 0, "mean": 0.0, "max_win": 0.0}
    w = min(window, n)
    win = nll.unfold(0, w, 1).mean(dim=1)   # mean NLL of every window of w tokens
    return {"n": n, "mean": nll.mean().item(), "max_win": win.max().item()}

Work in mean NLL (nats per token) rather than perplexity. The two are equivalent, since perplexity is exp(mean), but NLL is additive, easy to average over windows and does not overflow for gibberish. The function returns the three features worth having: token count, whole-prompt mean and the worst window.

Calibrating the threshold

Never copy a threshold from a paper or another product. It depends on the scorer model, the tokenizer, your users' language and the kinds of text they send. Instead, collect a sample of real benign prompts (thousands, not dozens), compute features, and set the threshold at a percentile that matches the false-positive rate you can tolerate:

import numpy as np

def calibrate(benign_texts, fpr=0.001, window=10):
    feats = [features(t, window) for t in benign_texts]
    max_wins = np.array([f["max_win"] for f in feats if f["n"] >= window])
    thr = float(np.quantile(max_wins, 1 - fpr))
    return {"window": window, "max_win_threshold": thr, "n_samples": len(max_wins)}

def is_suspicious(text, cfg):
    f = features(text, cfg["window"])
    if f["n"] < cfg["window"]:
        return False, f          # too short to judge; route to other checks
    return f["max_win"] > cfg["max_win_threshold"], f

At 0.1% FPR on a sample of 10,000 prompts, the threshold rests on the top 10 benign prompts, which is a noisy estimate, so use a larger sample or a less strict rate. Calibrate separately per segment type and per major language. Recalibrate when the scorer model or your traffic mix changes, and keep the benign sample under version control so the threshold can be reproduced.

Worked example

Take a support prompt: about 40 tokens of ordinary English asking how to reset a router. Suppose its per-token NLLs from GPT-2 range from about 0.5 to 6 nats, with a mean around 3.5, and the worst 10-token window, which contains a product model number, averages 5.2. These are illustrative values to show the arithmetic; run the code on your own data for real ones.

Now append a 20-token optimized suffix of mixed punctuation and word fragments. Each of those tokens is improbable, say 8 to 12 nats each. The whole-prompt mean rises from 3.5 to about 5.7 (140 nats plus 200 nats over 60 tokens), which might still sit under a whole-prompt threshold set loosely to avoid false positives. But every window that falls fully inside the suffix averages around 10 nats, far above a benign worst-window threshold of, say, 7. The windowed filter catches the attack that the mean nearly misses.

Now consider a benign false positive: a user pastes a 40-character API error code and a base64 blob. The base64 window is as surprising as the suffix. The fix is not a higher threshold, which would let attacks through, but segmenting: detect base64, hex, code blocks and URLs, and score only the natural-language segments with this filter.

False positives

The classifier approach from Alon and Kamfonas generalizes this: feed perplexity features together with length, segment type and language into a small gradient-boosted model, trained on benign traffic plus known attacks. It handles interactions, such as "short and surprising is normal", that a single threshold cannot.

Bypasses and where the filter fits

The filter's weakness is fundamental. It detects text that is unlike natural language, and many attacks are natural language. Hand-written jailbreaks, role-play framings, multi-turn escalation and indirect prompt injection in documents are all fluent, so they pass. Automated attacks have also adapted. AutoDAN (Liu and colleagues, 2023) uses a genetic algorithm to produce readable jailbreak prompts, and Jain and colleagues showed that adding a perplexity term to the GCG objective lets the optimizer trade fluency against attack strength. In their experiments that made the attack weaker, which is a real benefit, but it shows a determined attacker can get under the threshold.

So place the filter where its strength matters: as an early, cheap layer that removes the optimized-gibberish class, alongside other defenses: prompt injection scanners for fluent injected instructions, input transformations such as paraphrasing, output-side checks, and the broader controls in jailbreak defense. Apply it to retrieved chunks as well as user prompts, because a suffix can be planted in a document. The background on how suffixes are found is in GCG universal adversarial attacks.

Operating it

• Latency. GPT-2 small on a modern CPU scores a few hundred tokens in tens of milliseconds; batch requests on a GPU at high volume. Measure on your own hardware before committing to a latency budget.
• Fail mode. Decide in advance what happens when the scorer is down. For a public chatbot, fail open with logging; for an agent with tool access, fail to a stricter path.
• Action on flag. Blocking outright frustrates users with false positives. A softer response is to strip the high-NLL span and ask the user to rephrase, or to route the request to a model configuration with stricter output checks.
• Logging. Record features and the decision, not just a boolean, so you can re-tune thresholds against flagged traffic and catch drift.
• Adversarial testing. Keep a corpus of known suffixes and fluent jailbreaks in CI and track the filter's detection rate on each class separately.

Trade-offs

A perplexity filter is cheap, model-agnostic and needs no attack-specific training. Against unadapted GCG-style suffixes it is effective. It costs a small model in the request path, it produces false positives on exactly the inputs technical users send most (code, logs, encoded data, other languages), and it gives almost no protection against fluent attacks. Windowing improves detection of short suffixes at the price of more false positives on short unusual spans. A learned classifier on perplexity features reduces false positives but needs labeled data and maintenance. Use it as one layer, never the only one.

What to do next

1. Sample a few thousand real benign prompts and compute token count, mean NLL and max-window NLL with the code above.
2. Add segmenting for code, base64, URLs and non-Latin scripts before scoring.
3. Set a max-window threshold at a percentile that fits your false-positive budget, per segment type.
4. Run the filter in log-only mode for a week, review what it flags, then turn on enforcement.
5. Apply the same check to retrieved chunks and tool outputs, not just user prompts.
6. Keep a CI corpus of suffix attacks and fluent jailbreaks, and report detection per class.
7. Keep learning: GCG attacks, perplexity, injection scanners and jailbreak defense.