Why LLM Abuse and Fraud Prevention Needs a Threat Model

Start with the business outcome

A support assistant can produce an offensive answer, disclose a private order, authorize a fraudulent refund, or exhaust the inference budget. Those outcomes have different causes and require different evidence. Calling all of them a jailbreak obscures the control that failed. Abuse and fraud prevention, abbreviated AFP in this series, means protecting people, data, money, and service availability across the whole application.

Consider a fictional retailer, Northstar, whose assistant searches policies and prepares refunds. The useful business goal is faster resolution of valid claims. The security goal is that the assistant cannot expand a customer's authority. The operational goal is that suspicious workloads do not consume the capacity needed by legitimate customers. A complete design evaluates all three rather than optimizing the model's refusal rate alone.

Draw five boundaries

Separate the customer interface, orchestration runtime, retrieval service, action service, and external processor. Record the identity and data crossing each edge. A conversation identifier is useful for correlation but is not proof that its caller owns an order. A retrieved document can describe policy but cannot change the active policy version. A generated tool argument is a proposal whose legitimacy remains to be checked.

At each edge, ask who can modify the input and who bears the consequence. A supplier may control an invoice, an employee may edit a knowledge article, and a customer may upload a screenshot. These actors are not equally trusted for every operation. Trust should be attached to a particular claim or capability, not to a whole document merely because it came from inside the company.

Build an abuse-case register

For each row, name an owner and write a testable invariant. “Protect privacy” is not yet an invariant. “A request authenticated as tenant A cannot retrieve records labeled tenant B” is. State the behavior for unavailable dependencies and expired evidence. An authorization service timeout must not silently turn into a successful permission decision.

Include insiders and compromised integrations without assuming every unusual request is hostile. Threat modeling describes what the system must withstand, not an accusation against a particular user. A legitimate analyst may quote a malicious instruction while investigating an incident; your policy should preserve that workflow where possible.

Separate likelihood from consequence

A weak content classifier and a powerful payment tool create a different risk from the same classifier in a read-only demo. Estimate impact using the maximum reachable action, affected population, and recovery cost. Estimate exposure using who can submit inputs, how often, and what evidence would reveal misuse. These are planning assumptions; do not present guessed numbers as incident statistics.

For Northstar, a payment limit constrains loss even when detection fails. Per-user document access constrains disclosure even when the model follows an injected instruction. Reducing the amount of authority available to the assistant often removes a whole outcome from the threat model. This is why tool design belongs in AFP planning before another prompt filter is purchased.

Define a decision contract

Illustrative python; adapt the persistence and identity adapters to your service.

# Policy result carried between services, not model-generated authority.
decision = {
    "outcome": "review",  # allow, deny, review, unavailable
    "policy_version": "refund-policy-7",
    "reason_code": "destination_changed",
    "evidence_ids": ["order-fixture-42"],
    "action_digest": "server-computed-digest",
    "expires_at": "2026-10-15T12:00:00Z",
}
assert decision["outcome"] != "allow"  # No execution on review.

Keep these fields in a server-owned record. The digest binds the decision to an exact proposal; it is not an authentication mechanism by itself. Sign or protect the record through your normal service trust model, enforce expiry, and verify the caller when retrieving it. Returning a JSON object from a model that happens to contain “allow” must never create this record.

Versioned reasons help engineering and operations speak the same language. They also let you distinguish a deliberate denial from a guardrail outage. Avoid exposing detailed detector internals to the requester, but provide enough information for legitimate users to correct missing evidence or request review.

Prioritize controls and test ownership

Implement access control and transaction constraints first because they govern concrete effects. Add input and output screening for the harmful content categories relevant to the product. Introduce behavioral detection for repeated misuse and review for uncertain high-impact cases. Each control should have a named maintainer, a latency budget, a failure policy, and a regression suite.

Run a tabletop exercise in which a malicious document reaches the model and the model proposes an unauthorized refund. The expected outcome is that the refund service rejects it, the user sees no misleading success message, and an operator can reconstruct the decision. Then repeat with the detector unavailable and with a legitimate refund. A design that blocks every case is contained but not useful.

Worked design review: a refund assistant gains email access

The initial Northstar prototype can read an order and prepare a refund proposal. A product request adds email so the assistant can send a receipt. The obvious implementation is to attach a general mailbox tool. During review, the team discovers that the tool can search the entire mailbox, download attachments, and send to arbitrary recipients. The requested feature was one receipt; the proposed authority includes unrelated customer correspondence and unrestricted delivery.

Rewrite the capability as send_refund_receipt(operation_id). The backend loads the committed operation, resolves the customer contact from the order, renders an approved receipt template, and sends at most once. The assistant cannot choose the recipient, amount, or attachment. A separate read-only status operation returns whether delivery succeeded. This design removes several attack outcomes instead of trying to recognize every malicious instruction that might trigger them.

The review then considers a compromised order record. If the customer contact changed recently, should the receipt go to the new address? That is a product and identity-policy question. Resolve it explicitly using the authoritative contact-change workflow. Do not let the assistant decide from the conversation. Record the chosen rule and add a fixture where the destination changes between refund completion and receipt generation.

Next, consider an outage after the delivery provider accepts the message but before the service stores the result. The operation identifier must let the service reconcile or deduplicate delivery. Otherwise the assistant's ordinary retry can create duplicate messages. The abuse model therefore includes both malicious redirection and accidental repetition; the same narrow capability helps contain both.

The acceptance record names the owner of the receipt service, the allowed data fields, the maximum recipient count, the retention of delivery evidence, and the pause control. A test submits a forged recipient argument and expects rejection because the tool schema does not accept it. Another test calls the backend directly with an operation belonging to another tenant and expects denial. A third replays the same operation and expects one logical delivery.

This review produces a concrete feature rather than a vague requirement to add more moderation. The model still needs suitable content behavior, but receipt authority is now defined by an operation record and a narrow service contract. The resulting threat model is smaller, the tests are more precise, and incident containment can target receipt delivery without disabling order lookup.

Decide what evidence permits launch

Launch evidence should combine successful legitimate journeys, failed abuse attempts, and demonstrated recovery. Record the evaluated model and SDK versions, tool inventory, policy configuration, dataset scope, and unresolved risks. Require owners to accept specific residual risks rather than signing a generic statement that the assistant is safe.

The NIST AI RMF Playbook provides a broader risk-management vocabulary; the register and Northstar workflow here are original implementation examples. Start small: one protected action, one clear invariant, and one incident drill. Expand capabilities only when their additional trust boundaries have corresponding controls and evidence.

Practice and continue

Open the interactive companion lab to inspect the assumptions and outcomes with synthetic data. Then return to the 25-chapter AFP learning path.

• How LLM Jailbreaks Happen: Instruction Confusion and Context Pressure

Sources and implementation scope

Primary references checked October 15, 2026. Framework details follow the linked documentation; the Northstar cases, policy contracts, and test designs are original illustrative designs. Python snippets labeled illustrative are not complete production integrations. ADK package metadata checked for this series reported 2.12.0; pin and test the version used by your application.

• NIST AI RMF Playbook
• OWASP excessive agency, 2025 edition