ShieldGemma, in depth: one policy per pass, Yes/No logit scoring, threshold calibration and where it fits beside Llama Guard

What ShieldGemma is

The original ShieldGemma, released in July 2024, comes in three sizes, 2B, 9B and 27B, all fine-tuned from Gemma 2. They classify text: either a user prompt on its own, or a model response together with the prompt that produced it. ShieldGemma 2, released in 2025, is a separate 4B model built on Gemma 3 that classifies images against three policies: sexually explicit, dangerous content and violence or gore. This article focuses on the text models; the image model uses the same scoring idea with an image and a policy as input.

What ShieldGemma is not: a prompt-injection detector, a jailbreak detector or a factuality checker. It judges content against harm policies. A polite prompt that smuggles instructions into a tool call violates none of its policies. Pair it with an injection classifier and with output controls if those threats matter.

The four harm policies

The text models were trained and evaluated on four harm types. The model card phrases each one twice, once for prompt classification ("The prompt shall not contain or seek generation of content that ...") and once for response classification ("The chatbot shall not generate content that ..."). Use the wording from the card; the model is sensitive to it.

Each request is classified against one policy at a time. Four policies means four scores, and your policy engine decides how to combine them.

The prompt template

The prompt-classification template, as published on the model card, has four parts: a preamble casting the model as a policy expert, the user text wrapped in Gemma's <start_of_turn> and <end_of_turn> control tokens, the guideline, and an epilogue that demands an answer starting with Yes or No:

You are a policy expert trying to help determine whether a user prompt is in
violation of the defined safety policies.

<start_of_turn>
Human Question: {user_prompt}
<end_of_turn>

Our safety principle is defined in the below:

* "No Hate Speech": The prompt shall not contain or seek generation of content
  that expresses, incites, or promotes hate based on race, gender, ethnicity,
  religion, nationality, sexual orientation, disability status, or caste.

===

Does the human question violate the above principle? Your answer must start with
'Yes' or 'No'. And then walk through step by step to be sure we answer
correctly.

Response classification uses the same structure with a preamble about judging an AI response, a second wrapped turn holding the chatbot response after the human question, the response-side guideline, and an epilogue asking whether the chatbot response violates it. Copy that template from the model card rather than reconstructing it.

Note the order: the user text comes before the guideline. That detail, which looks cosmetic, is what makes multi-policy scoring cheap, as the cost section shows. Also escape or strip control tokens from user text before inserting it: a user who types <end_of_turn> can otherwise close the wrapped turn early and append their own instructions to the classifier.

Scoring from two logits

The model is used in scoring mode. You never generate the step-by-step explanation the epilogue mentions. You run one forward pass over the prompt, take the logits at the last position for the tokens Yes and No, and softmax just those two. P(Yes) is the violation score.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("google/shieldgemma-2b")
model = AutoModelForCausalLM.from_pretrained(
    "google/shieldgemma-2b", torch_dtype=torch.bfloat16, device_map="auto")
YES, NO = tok.get_vocab()["Yes"], tok.get_vocab()["No"]

# The model card template, with the guideline bullet replaced by a placeholder.
TEMPLATE = """You are a policy expert trying to help determine whether a user prompt is in
violation of the defined safety policies.

<start_of_turn>
Human Question: {user_prompt}
<end_of_turn>

Our safety principle is defined in the below:

* {guideline}

===

Does the human question violate the above principle? Your answer must start with
'Yes' or 'No'. And then walk through step by step to be sure we answer
correctly.
"""

CONTROL = ("<start_of_turn>", "<end_of_turn>")

def clean(text):
    # Repeat until stable: one pass per token can be defeated by nesting,
    # e.g. "<start_of_<end_of_turn>turn>" collapses into "<start_of_turn>".
    while any(t in text for t in CONTROL):
        for t in CONTROL:
            text = text.replace(t, "")
    return text.strip()

@torch.no_grad()
def score(user_prompt, guideline):
    prompt = TEMPLATE.format(user_prompt=clean(user_prompt), guideline=guideline)
    inputs = tok(prompt, return_tensors="pt").to(model.device)
    logits = model(**inputs).logits[0, -1, [YES, NO]]
    return torch.softmax(logits.float(), dim=0)[0].item()   # P(Yes)

Three details matter. Restricting the softmax to two tokens makes the score independent of whatever else the model might have said. The tokens must be the exact vocabulary entries the model card uses; a leading-space variant is a different token. And the model repository is gated on Hugging Face, so accept the Gemma terms with the account your deployment uses before the first download fails in production.

Where it sits in the request path

In a typical deployment the prompt-side check runs before the main model, and the response-side check runs on the completed or partial response. Scores go to a policy engine with a threshold per policy, rather than a single global cutoff, because the cost of a miss differs by policy. Borderline scores, for example within 0.1 of the threshold, are logged for review and become your calibration data. For streamed responses, score accumulated text at intervals rather than per token; streaming moderation describes the release-watermark pattern that keeps unscored text from reaching the user.

Cost: one pass per policy, shared prefix

One policy per pass sounds expensive: four policies, four forward passes, then four more on the response. But every pass is prefill only, with no decoding, and the template puts the preamble and the user text first. All four policy prompts for one request share that prefix and differ only in the last hundred or so tokens. With a serving engine that caches prefixes, such as vLLM with automatic prefix caching, the shared part is computed once and each extra policy costs only its guideline and epilogue tokens.

A rough FLOP budget makes this concrete. Prefill costs about two FLOPs per parameter per token. The 2B model has roughly 2.6 billion parameters, so a 400-token prompt costs about 2 x 2.6e9 x 400, or 2.1 TFLOP. Four policies without prefix sharing cost 8.3 TFLOP; with a shared 300-token prefix and 100 tokens per policy they cost 2 x 2.6e9 x (300 + 4 x 100), about 3.6 TFLOP, a 2.3x saving. On a modern data-centre GPU either is milliseconds, but at thousands of requests per second the factor decides how many GPUs the guard needs.

Batch the four policies of a request together, and batch across requests. Size choice is the bigger lever: the 27B model costs about ten times as much per token as the 2B, and as the next sections show, it is not uniformly more accurate.

Worked example: calibrating a threshold

Published results use optimal F1, which picks the best threshold for each dataset after seeing its labels. You do not get that threshold for free. Calibrate on your own traffic: sample a few thousand real requests, label them against your policy, score them, and choose the threshold that meets your recall target.

import numpy as np

def pick_threshold(scores, labels, min_recall=0.90):
    scores, labels = np.asarray(scores), np.asarray(labels, bool)
    best = None
    for t in np.unique(scores)[::-1]:          # high to low
        flagged = scores >= t
        recall = (flagged & labels).sum() / max(labels.sum(), 1)
        if recall >= min_recall:
            precision = (flagged & labels).sum() / max(flagged.sum(), 1)
            best = (float(t), float(recall), float(precision))
            break
    return best   # highest threshold reaching the recall target

Base rates dominate the outcome. Suppose 2 percent of 10,000 daily prompts truly violate a policy and your chosen threshold gives 85 percent recall with a 3 percent false-positive rate. You catch 170 of the 200 violations, and you also flag 294 of the 9,800 benign prompts. Of the 464 flags, only 37 percent are real. Across four independent policies each at 1 percent false positives, about 3.9 percent of benign traffic trips at least one. Decide whether a flag blocks, warns or routes to review with those numbers in front of you, and re-calibrate when traffic changes.

Reading the published benchmarks

These are optimal F1 / AU-PRC from the model card; SG Prompt and SG Response are Google's internal evaluation sets. Three readings follow. The sizes are close: the 2B model is within about two points of the 27B on most columns. Bigger is not uniformly better: on OpenAI Mod the 9B scores 0.821 F1 against 0.805 for the 27B, and on ToxicChat F1 the 9B trails the 2B. And no column is your traffic. Start with the 2B or 9B, calibrate both on your own labelled sample, and move up only if the larger model wins there.

Custom policies

Because the guideline is plain text in the prompt, you can write your own: a policy for medical dosing advice, for a children's product, for a brand's tone. This is the main attraction and the main risk. The model card warns that ShieldGemma is highly sensitive to how a safety principle is described, and it was trained on the four policies above, so a custom guideline is a zero-shot use. Write it in the same "shall not contain or seek generation of content that" form, keep it to one concern, include concrete examples, and evaluate it on a labelled set exactly as you would a new classifier. If a custom policy matters to your business, a fine-tuned classifier or a taxonomy-driven model such as Llama Guard with custom categories may serve better.

Failure modes

• Control-token injection. User text containing Gemma turn markers breaks out of the wrapped turn. Strip or escape them before templating.
• Wrong token ids. Using a space-prefixed Yes, or the chat template's turn token, gives meaningless scores. Assert the ids at start-up and test with known violating and benign prompts.
• Untuned thresholds. Copying 0.5 or a benchmark-optimal value produces either a leaky guard or a wall of false positives. Calibrate per policy on real traffic.
• Policy drift by paraphrase. Editing a guideline's wording changes scores. Version guidelines like code and re-run the calibration set on every change.
• Language and domain gaps. Benchmarks are mostly English. Measure per-language recall before relying on it for multilingual traffic.
• Category blind spots. Fraud, privacy leaks, prompt injection and self-harm nuance outside the four policies get no score at all. Add classifiers for what your threat model actually contains.

Trade-offs

ShieldGemma is a strong default when you want a self-hosted, probability-scored check on the four common harm types and you can afford a calibration pass. Meta Purple Llama covers the competing tool set if you need injection and code checks from one vendor.

What to do next

1. Accept the Gemma terms, load the 2B model, and assert the Yes and No token ids at start-up.
2. Implement score() with control-token stripping and run ten known-violating and ten benign prompts through each of the four policies as a smoke test.
3. Label 2,000 or more real prompts and responses, score them, and choose a threshold per policy with pick_threshold at your recall target.
4. Compute the flag rate and precision at those thresholds using your real base rates, and decide which policies block and which route to review.
5. Enable prefix caching in your serving engine and batch the four policies per request; measure p95 guard latency under load.
6. Version every guideline string, and re-run calibration when you edit one or move to a larger model.