STRIDE for LLM Systems, in Depth

STRIDE in one table

Each STRIDE category is the negation of a security property. Naming the property is what turns a category into a design question, because it tells you which control you are missing.

The right-hand column is the whole point of this article: every row there is a real class of LLM incident, and each maps cleanly to one property. Prompt injection, the threat most people start with, does not get a letter of its own. It is a technique that usually produces Tampering (it alters the instruction stream) and is used to achieve Elevation of privilege or Information disclosure. Classifying it by outcome rather than by name is what makes the mitigations obvious.

STRIDE in one table

Each STRIDE category is the negation of a security property. Naming the property is what turns a category into a design question, because it tells you which control you are missing.

The right-hand column is the whole point of this article: every row there is a real class of LLM incident, and each maps cleanly to one property. Prompt injection, the threat most people start with, does not get a letter of its own. It is a technique that usually produces Tampering (it alters the instruction stream) and is used to achieve Elevation of privilege or Information disclosure. Classifying it by outcome rather than by name is what makes the mitigations obvious.

Why LLM systems strain the method

Three properties of LLM components break assumptions that classic STRIDE sessions lean on.

Instructions and data share one channel. In a web application, parsing enforces the boundary: SQL parameters cannot become SQL. In an LLM application, the system prompt, the user's message, a retrieved wiki page and an attacker's email arrive in one token sequence, and the model cannot authenticate which tokens came from whom. So never draw the model as a trust boundary. Boundaries belong where code can enforce them: around the orchestrator, the tool executor and the stores.

The model is a non-deterministic process. It does what its weights, context and sampling make likely, so assume any instruction-shaped text in the context may be followed. The model will refuse is a probabilistic claim, not a control.

Tools turn output into action. Once the model can send email or call an API, its output is a request issued with somebody's credentials. That is why Elevation of privilege dominates agent threat models, and why the key question is: whose authority does each tool call carry?

Drawing the system

STRIDE is applied to a data-flow diagram, and an LLM system needs more elements on that diagram than teams expect. Draw at least these:

• External entities: the end user, the authors of any content you ingest (email senders, web pages, uploaded files, issue comments), the model provider if you call a hosted API, and every downstream API a tool reaches.
• Processes: the orchestrator, model inference, the tool executor and any ingestion pipeline.
• Data stores: the retrieval index, agent memory, conversation history, prompt templates, model weights or adapters you host, and logs and traces.
• Data flows: each arrow, labelled with what it carries. The arrow from the orchestrator to the model should be labelled with every source that can contribute tokens to it.

Microsoft's STRIDE-per-element convention says which letters to ask about for each element type. External entities can be spoofed and can repudiate. Processes are exposed to all six categories. Data stores face Tampering, Information disclosure and Denial of service, plus Repudiation when the store is an audit log. Data flows face Tampering, Information disclosure and Denial of service. An element with no entry for an applicable letter is a hole in the analysis, not evidence of safety.

The six letters, applied to LLM components

Spoofing. Can any component be fooled about who is speaking? Users are authenticated by the orchestrator as usual. The LLM-specific cases are content that claims an identity, such as a retrieved document formatted as a system message or a tool result impersonating the user, and a malicious tool server posing as a trusted one. Authenticate and pin tool servers, and record the provenance of every context segment in your own data structures, so policy uses provenance rather than what the text claims.

Tampering. What can change instructions, data or artifacts without authorisation? Indirect prompt injection tampers with the instruction stream through any ingestion path. Writes to the retrieval index or agent memory by a low-privilege party tamper with a store that later steers a higher-privilege session. Hosted weights, adapters and prompt templates are artifacts: verify them with hashes and signed deployments, like binaries.

Repudiation. Could you reconstruct who caused an action and why? Outputs are sampled, so the user's message alone cannot replay an incident. Log the assembled context or references to its parts, the model version, sampling parameters, every tool call with arguments and result, and the acting identity. Those logs hold user data, which creates a disclosure obligation on the log store.

Information disclosure. What can the model see that the requester should not, and what can carry it out? Filter retrieval by the requester's permissions before ranking, not after generation. Assume the system prompt will leak and keep secrets out of it. Treat output that fetches on render, such as markdown images, as an exfiltration channel, and remember shared caches and unprotected embeddings.

Denial of service. What can an attacker make expensive? Cost scales with context and output length, agents can loop on a failing tool, and one shared provider key lets one tenant exhaust everyone's rate limit or budget. Budget per request, user and tenant, cap agent steps, and fail closed.

Elevation of privilege. Can the model cause an action the user could not perform directly? If the tool executor holds a service account broader than the user, every successful injection is an escalation: the confused deputy. Run tools with the user's scoped credentials, allowlist tools per task, confirm consequential actions, and sandbox code execution without ambient credentials.

Worked example: an inbox triage agent

Consider an inbox triage agent, the system in the diagram above. It reads a user's email, searches the company wiki, drafts replies, can send email for the user and can open CRM tickets. Email senders are anyone on the internet. A real session yields more rows; these are the ones that change the design.

Look at where the mitigations land: none of them asks the model to behave. Each is ordinary engineering at a boundary your code owns, namely delegated credentials, confirmation gates, output sanitising, query-time access filtering, budgets and logging. That is the most useful thing STRIDE teaches about LLM systems.

Threats as code

A threat model in a document goes stale the first time someone adds a tool. Keep it next to the code and let CI check that every element was considered for every applicable letter; a not-applicable needs a reason.

# threat_model.py -- checked in CI
APPLICABLE = {            # STRIDE-per-element
    "external": "SR",
    "process":  "STRIDE",
    "store":    "TID",
    "log":      "TRID",   # audit stores also answer for repudiation
    "flow":     "TID",
}

ELEMENTS = {
    "user":            "external",
    "email_senders":   "external",
    "orchestrator":    "process",
    "model":           "process",
    "tool_executor":   "process",
    "retrieval_index": "store",
    "agent_memory":    "store",
    "logs":            "log",
    "email_to_context": "flow",
    "context_to_model": "flow",
    "tool_to_mail_api": "flow",
}

THREATS = [
    {"id": "T1", "element": "email_to_context", "letter": "T", "status": "mitigated",
     "control": "untrusted-content label; confirm new external recipients"},
    {"id": "T2", "element": "tool_executor", "letter": "E", "status": "mitigated",
     "control": "per-user delegated OAuth token; reply-in-thread only"},
    {"id": "NA1", "element": "email_senders", "letter": "R", "status": "n/a",
     "reason": "senders are untrusted by definition; we never rely on their claims"},
    # ... one entry or explicit n/a per element and applicable letter
]

def gaps():
    seen = {(t["element"], t["letter"]) for t in THREATS}
    missing = []
    for name, kind in ELEMENTS.items():
        for letter in APPLICABLE[kind]:
            if (name, letter) not in seen:
                missing.append(f"{name}:{letter}")
    bad_na = [t["id"] for t in THREATS if t["status"] == "n/a" and not t.get("reason")]
    return missing, bad_na

if __name__ == "__main__":
    missing, bad_na = gaps()
    if missing or bad_na:
        raise SystemExit(f"unmodelled: {missing}; n/a without reason: {bad_na}")

Add a review rule: any change that adds a tool, data source, memory feature or output channel must add its element to ELEMENTS, so the checker forces the threat discussion exactly when the surface grows.

Where STRIDE does not fit

STRIDE asks about violations of security properties. Several important LLM risks are not violations of any of the six, and forcing them into a letter produces confused analysis.

• Hallucination. A confident wrong answer with no attacker involved is a reliability and safety problem, not Tampering. It needs evaluation, grounding and citation checks, not access control. It becomes a STRIDE concern only when someone can exploit it, for example by registering a package name that a model tends to invent.
• Harmful or policy-violating content. Whether outputs are appropriate is a content-safety question with its own taxonomies and classifiers.
• Training-time threats. Poisoning and backdoored weights are Tampering upstream of your diagram. If you fine-tune, add the training pipeline; otherwise record them as supply-chain assumptions.
• Excessive autonomy. An authorised but unwanted action is a product-design failure; STRIDE flags the permission, not which actions need a human.

Use STRIDE for the security properties and run the other lenses beside it. Do not stretch the mnemonic until every risk fits, because a category that means everything stops prompting the right question.

Running it in practice

Run the first session with the engineer who wrote the orchestrator, the owner of the tool integrations and someone from security; ninety minutes per major feature is typical. Start from the diagram, not a list of famous attacks. Record threats even when you believe they are mitigated, with the control, because that record is what you test against.

Then turn mitigations into tests: an injection fixture in the email corpus that tries to send to an external address and must hit the confirmation gate; a retrieval test with a user who lacks access to a seeded page; a budget test that drives a looping tool and expects termination. Re-run them when the model changes, since an upgrade shifts every model-dependent probability. Revisit the threat model on triggers, not a calendar: a new tool, ingestion source, credential, memory feature, rendering surface or model.

Failure modes

Threat modelling LLM systems fails in recognisable ways.

• Treating the system prompt as a control. Instructions such as never reveal this or ignore instructions in documents reduce the success rate of some attacks. They are not a boundary and should never be the only entry in the mitigation column.
• Drawing the model inside the trust boundary. If the diagram shows the model as trusted, every injection path disappears from the analysis.
• Analysing only the chat box. The user's message is the least dangerous input. Ingested documents, tool results and memory written by other sessions are where hostile text arrives.
• No residual risk. Some threats, such as injection reaching the model at all, cannot be eliminated, only contained. Write down the containment and the residual risk explicitly, so nobody later believes the problem was solved.

What to do next

Go deeper with data-flow diagrams for LLM systems, which covers the drawing step in detail, and threat modelling LLM systems, which ranks findings with the lethal trifecta lens. The general practice is in threat modelling as a practice. Two mitigations from the example have their own articles: agent authorization and the confused deputy and egress control for agents.

1. Draw a data-flow diagram of one LLM feature, including every source that can contribute tokens to the context.
2. Put trust boundaries around the orchestrator, tool executor and stores, never around the model.
3. Walk every element with the STRIDE-per-element table and record a threat or a reasoned not-applicable for each letter.
4. For every tool, write down whose credentials it uses; replace any service account broader than the user.
5. Encode the model as code with the coverage checker and add it to CI.
6. Turn each mitigated threat into an automated test, and re-run the suite on every model or tool change.