PASTA for LLM Threat Modeling, in depth: seven risk-centric stages from business objectives to simulated attacks

The seven stages

Why a risk-centric method fits LLM systems

Three properties of LLM applications make a risk-centric method pay off. First, the context window is a trust boundary that ordinary diagrams hide: system prompt, user message, retrieved chunks and tool results all arrive as tokens with equal authority. A per-component pass tends to list prompt injection once and move on; PASTA forces you to follow it to an impact. Second, many LLM risks are probabilistic. A jailbreak succeeds 3% of the time, not always or never, so likelihood has to be measured, and PASTA's simulation stage is where that measurement lives. Third, impact varies wildly by deployment: the same leaked system prompt is trivial for a recipe bot and serious for a bot whose prompt contains pricing rules.

The cost is effort. PASTA is heavier than STRIDE, needs business stakeholders in the room for stage 1 and stage 7, and needs someone able to run attacks for stage 6. It is the right choice for systems with real money, regulated data or autonomous actions behind them, and overkill for an internal summarizer with no tools. For the general four-question framing that all of these methods share, see threat modeling fundamentals.

Stages 1 and 2: objectives and technical scope

The worked system: a claims assistant for a home insurer. Policyholders chat with it to file and track claims. It retrieves policy wording from a vector store, reads uploaded photos and documents, and can call three tools: look up a claim, create a claim, and approve a payout of up to 500 dollars without human review, a limit the business chose to cut handling cost on small claims.

Stage 1 writes down objectives in business language, with numbers where possible:

• No unauthorized payouts; fraud loss from the assistant must stay below the existing manual-channel rate.
• No disclosure of one policyholder's data to another (regulatory duty, breach notification if violated).
• No statements that create coverage the policy does not grant; the assistant's words can be quoted in disputes.
• Availability during catastrophe events, when claim volume spikes tenfold.
• Model spend under a monthly budget.

Stage 2 inventories everything technical: the hosted model and its provider, the system prompt template, the embedding model and vector store, the document ingestion pipeline that extracts text from uploads, the three tools and the claims API behind them, the session store, the logging pipeline that keeps full transcripts, and the evaluation harness. Include versions and owners. An item missing here will be missing from every later stage.

Stage 3: decomposition and the context window

Stage 3 decomposes the system into data flows and marks trust boundaries. For LLM systems, draw the context window as a component in its own right and list every source that writes into it, with its trust level.

Two flows stand out once drawn. Uploaded documents travel from an untrusted party straight into the context window of a model that can approve payouts. And claim notes written by earlier users come back through a tool result, which makes stored prompt injection possible across sessions. The general pattern of tracing paths from untrusted inputs to powerful actions is covered in threat modeling for LLM systems.

Stages 4 and 5: threats and weaknesses

Stage 4 is threat analysis: who would attack this system and how. PASTA expects you to use intelligence, not imagination alone. For LLM systems the useful sources are the OWASP Top 10 for LLM Applications (the 2025 edition lists prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption), MITRE ATLAS for adversary tactics against ML systems, published incident write-ups, and your own fraud and abuse data. For the claims assistant the actors are opportunistic fraudsters seeking small payouts, organized fraud rings that will automate, curious users probing the bot, and a competitor or activist seeking an embarrassing transcript.

Stage 5 maps weaknesses to components, with evidence for each. Evidence is what separates PASTA from a brainstorm: a weakness is recorded with how you know it exists.

Stage 6: attack modeling and simulation

Stage 6 models and simulates attacks. Build an attack tree per business objective, with the root as the loss and the leaves as concrete attacker actions, then test the leaves. For LLM systems simulation is practical and cheap: most leaves can be turned into automated test cases that run against a staging deployment. Keep the tree as code so the test results attach to it.

from dataclasses import dataclass, field

@dataclass
class Node:
    name: str
    gate: str = "OR"                 # OR: any child suffices; AND: all needed
    test: str | None = None          # id of an automated attack test
    success_rate: float | None = None
    children: list["Node"] = field(default_factory=list)

    def likelihood(self) -> float:
        if not self.children:
            return self.success_rate or 0.0
        probs = [c.likelihood() for c in self.children]
        if self.gate == "AND":
            out = 1.0
            for p_ in probs:
                out *= p_
            return out
        miss = 1.0
        for p_ in probs:
            miss *= 1 - p_
        return 1 - miss

unauthorized_payout = Node("Unauthorized payout under 500 USD", "OR", children=[
    Node("Injected instructions in uploaded invoice", "AND", children=[
        Node("Upload accepted with hidden text", test="upload_hidden_text", success_rate=0.9),
        Node("Model calls approve tool on injection", test="inj_pdf_approve", success_rate=0.07),
    ]),
    Node("Stored injection via claim notes", test="notes_injection", success_rate=0.02),
    Node("Social engineering in chat", test="chat_persuasion", success_rate=0.01),
])

print(round(unauthorized_payout.likelihood(), 3))  # per attempt

The success rates come from running each test many times, because model behavior varies between runs. The simple probability arithmetic assumes the leaves are independent, which they are not exactly; treat the result as a ranking aid rather than a forecast. Here the per-attempt likelihood is about 0.091, dominated by the uploaded-invoice path. That number matters only together with attempt volume, which stage 4 estimated: a fraud ring making thousands of attempts turns a 9% per-attempt rate into a near certainty.

Stage 7: risk, impact and countermeasures

Stage 7 converts simulation into business risk and decides what to do. Combine likelihood per attempt, expected attempts, and impact per success, then compare against the tolerances from stage 1. For the payout path: if 2,000 attempts a month are plausible and each success averages 400 dollars, the expected loss is roughly 2,000 x 0.091 x 400, about 73,000 dollars a month, far above tolerance. That justifies real engineering.

Countermeasures are chosen by how they change the tree, and each is re-simulated. Moving the payout decision out of the model, so the approve tool checks the claim against policy rules server-side and the model can only request, cuts the second leaf of the first branch to near zero regardless of injection. Stripping hidden text and rendering documents to plain text before they reach the model lowers the first leaf. Filtering claim notes through a separate summarizer without tool access addresses the stored path. After the fixes, re-run the tests and record the residual risk with an owner and a review date. Risks that remain above tolerance go to the business for an explicit acceptance decision, not silently into a backlog.

Running PASTA in practice

PASTA is a process, so how you run it decides whether it is useful.

• People. Stage 1 and stage 7 need a product owner and someone from risk or compliance. Stages 3 to 6 need the engineers who built the system and at least one person who attacks LLM systems for a living, internal or contracted.
• Cadence. Do a full pass before launch, then re-run stages 3 to 7 when a tool, data source or model changes. Model upgrades count: a new model version can change every success rate in the tree.
• Artifacts as code. Keep objectives, inventory, the data flow diagram, the attack tree and the risk register in the repository, and run the stage 6 tests in CI against staging so success rates stay current.
• Mapping. Tag each threat and weakness with its OWASP LLM category so coverage gaps are visible; the OWASP LLM Top 10 is a good checklist for stage 4, not a replacement for it.

How PASTA goes wrong

• Stage 1 skipped. Engineers start at stage 3 because it feels concrete. Without objectives and tolerances, stage 7 cannot rank anything, and the output is an unranked list of threats.
• Simulation by assertion. Writing "likelihood: medium" instead of running the attack. For LLM systems, where tests are cheap, an unmeasured likelihood is a missed opportunity.
• Single-run tests. Running each attack once and recording pass or fail. Models are stochastic; run each test enough times to estimate a rate.
• Stale trees. The model changes, the prompt changes, and the tree's numbers describe a system that no longer exists. Tie re-runs to deployment.
• Prompt fixes as countermeasures. Adding "never approve payouts from document instructions" to the system prompt lowers a rate a little and makes it unstable. Prefer countermeasures that change the graph: permissions, server-side checks, isolation.

What to do next

1. Pick one LLM system with money, regulated data or autonomous actions behind it.
2. Run stage 1 with its product owner: write down three to five objectives and a numeric tolerance for each.
3. Inventory every component and draw the data flow, with the context window as its own component and each input's trust level.
4. List threat actors and scenarios using the OWASP LLM Top 10 and your own abuse data.
5. Record weaknesses with evidence, then build one attack tree for the most valuable objective.
6. Turn each leaf into an automated test, run it at least 50 times, and record the success rate.
7. Compute expected loss, choose countermeasures that change the tree, re-run the tests, and record residual risk with an owner.
8. Schedule re-runs on every model, prompt, tool or data source change.