Why architecture matters here
Output handling matters because LLM output flows into other systems, and treating it as trusted creates injection vulnerabilities -- so handling it as untrusted (like any untrusted input to those systems) is essential, yet often overlooked. LLM output is text the model generated -- and it commonly flows into sinks (rendered as HTML, used in queries, passed to shells, executed as code, sent to APIs or other LLMs). If treated as trusted (used directly without safe handling), the output can attack the sink (XSS if it contains HTML/JS rendered in a browser; SQL injection if it's used in a query; command injection if passed to a shell; etc.) -- classic injection vulnerabilities, but originating from the LLM output. This is a serious, often-overlooked threat (much focus is on the input/prompt-injection side, but the output side is equally important) -- and it's exacerbated because LLM output is attacker-influenceable (via prompt injection -- an attacker manipulating the model into producing malicious output). So handling LLM output as untrusted (safely, for its sink) is essential to secure LLM applications, and understanding it (the threat, the sink-aware handling) is understanding a crucial, often-neglected aspect of LLM security.
The output-is-untrusted-input insight is the crucial reframing, and it connects LLM security to classic injection defense. The key realization: LLM output flowing into a sink is untrusted input to that sink -- exactly like user input flowing into that sink. So the same defenses that apply to any untrusted input apply to LLM output: if the output is rendered as HTML, it must be HTML-encoded (like any untrusted content in HTML -- to prevent XSS); if it's used in a SQL query, it must be parameterized (like any untrusted SQL input -- to prevent SQL injection); if it's passed to a shell, it must be escaped (like any untrusted shell input -- to prevent command injection). This reframing (LLM output is untrusted input to its sink -- apply the sink's standard injection defenses) connects LLM output handling to well-established injection defense (the decades of knowledge about handling untrusted input for HTML, SQL, shell, etc.) -- so you don't need novel defenses, but to apply the existing ones to LLM output. The mistake is treating LLM output as different/trusted (the model is 'helpful', so its output is safe) -- but the output is untrusted (attacker-influenceable, and even benign output can contain content that's dangerous in a sink) -- so it needs the same handling as any untrusted input. Understanding that LLM output is untrusted input to its sink (apply the sink's standard injection defenses) is understanding the crucial reframing that makes output handling tractable.
And the context-aware-encoding reality is the crucial technical detail, because the right handling depends entirely on the sink. There's no single 'sanitize LLM output' -- the safe handling is context-aware (specific to the sink the output flows into). Different sinks need different handling: HTML display -- HTML-encode (escape the HTML special characters -- so the output is displayed as text, not interpreted as HTML/JS -- preventing XSS); SQL -- parameterize (use parameterized queries -- so the output is data, not SQL -- preventing SQL injection); shell -- escape/avoid shell (don't pass output to a shell, or escape it properly -- preventing command injection); markdown -- sanitize the rendered markdown (markdown can contain HTML, data-leaking images, phishing links -- so sanitize it for safe rendering); structured/tool calls -- validate against a schema (the output structure validated, not trusted). The crucial point: you must apply the right encoding/handling for the specific sink (HTML-encoding doesn't prevent SQL injection; SQL parameterization doesn't prevent XSS) -- context-aware handling per the sink. Applying the wrong handling (or none) leaves the sink vulnerable. So safe output handling requires knowing where the output flows (its sinks) and applying the right handling for each (context-aware encoding/validation). Understanding the context-aware-encoding reality (the right handling depends on the sink -- apply per-sink defenses) is understanding the crucial technical detail of output handling.
The architecture: every piece explained
Top row: the threat and principle. The threat: LLM output flows into other systems (HTML, SQL, shell, code, APIs, other LLMs) -- where it can cause injection. Output is untrusted: the model's output may contain attacks (attacker-influenceable via prompt injection, or dangerous content in a sink) -- so it must be treated as untrusted. Encoding + escaping: context-aware output handling -- encoding/escaping the output for its specific sink (HTML-encode, SQL-parameterize, shell-escape) -- the safe handling. Injection sinks: where the output flows and can cause injection -- HTML (XSS), SQL (injection), shell (command injection), code (code injection) -- the sinks needing sink-specific handling.
Middle row: specific risks and validation. Markdown and links: LLM output is often rendered as markdown -- which can contain HTML (XSS), images (that leak data via the image URL -- e.g., an image pointing to an attacker's server with data in the URL), or phishing links -- rendered-output risks needing markdown sanitization. Tool-call validation: when the output is structured (a tool call -- the model requesting a tool with arguments), it must be validated (against a schema -- the tool and arguments checked, not trusted) -- so a malicious/malformed tool call is caught. Downstream injection: output used as input to other systems (including other LLMs -- e.g., a multi-agent system where one agent's output is another's input) -- carrying attacks downstream (the output injecting into the downstream system). Schema enforcement: constraining the output shape (forcing the output to a schema -- structured output -- so it's constrained, and validated against the schema) -- reducing the attack surface.
Bottom rows: relationship and display. vs prompt injection: output handling (the output side -- handling the model's output safely) is distinct from but related to prompt injection (the input side -- adversarial input manipulating the model) -- an injected prompt often manifests as a malicious output (so both sides matter -- defend the input against injection, and handle the output as untrusted). Human display safety: safely displaying output to humans -- preventing XSS (HTML-encoding), phishing links (checking/warning on links), and data-leaking images (sanitizing markdown images) -- the human-facing display risks. The ops strip: sink-aware encoding (applying the right encoding/handling for each sink the output flows into -- HTML, SQL, shell, markdown -- context-aware), validation (validating structured output -- tool calls, schemas -- against the expected structure), and monitoring (monitoring the output handling -- caught injections, anomalous outputs -- for detecting attacks and issues).
End-to-end flow
Trace an XSS via unhandled LLM output. An LLM-powered app displays the model's output in a web page (rendered as HTML). An attacker uses prompt injection (via a document the model processes) to make the model output malicious HTML/JavaScript (e.g., a script tag). If the app renders the output directly (treating it as trusted -- inserting it into the HTML without encoding), the malicious script executes in the user's browser (XSS -- the injected script running, potentially stealing the user's session, data, etc.). The vulnerability: the LLM output (untrusted -- attacker-influenced) was rendered as HTML without HTML-encoding (treated as trusted). The fix: HTML-encode the output before rendering (escaping the HTML special characters -- so the output is displayed as text, not interpreted as HTML/JS) -- so the malicious script is displayed as harmless text (not executed) -- the XSS prevented. Treating the LLM output as untrusted (HTML-encoding it for the HTML sink) prevented the XSS -- the context-aware handling (HTML-encoding for HTML display) applying the sink's standard injection defense to the LLM output.
The SQL and markdown vignettes show more sinks. A SQL case: an app uses the LLM's output in a SQL query (e.g., the model generates a query fragment, or the output is used as a query parameter). If the output is concatenated into the SQL (treated as trusted), a malicious output could inject SQL (SQL injection -- the output containing SQL that alters the query). The fix: parameterize (use the output as a bound parameter -- data, not SQL) or validate/constrain it -- so the output can't inject SQL (the SQL sink's standard defense applied to the LLM output). A markdown case: the app renders the LLM output as markdown. The output contains a markdown image pointing to an attacker's server with sensitive data in the URL (an image exfiltration -- when the markdown is rendered, the browser fetches the image, sending the data to the attacker). The fix: sanitize the markdown (restricting or checking image sources -- preventing the data-leaking image) -- handling the markdown rendering risk. The markdown sanitization prevented the exfiltration.
The tool-call and downstream vignettes complete it. A tool-call case: the LLM produces a structured tool call (requesting a tool with arguments). The app validates it against a schema (the tool must be an allowed tool, the arguments matching the expected schema and constraints) -- not trusting the tool call blindly (a malicious/malformed tool call -- e.g., requesting a disallowed tool or dangerous arguments -- caught by the validation) -- the schema validation handling the structured output safely. A downstream case: in a multi-agent system, one agent's output becomes another agent's input. The output is handled as untrusted for the downstream agent (since it could carry an injection -- the first agent's output attacking the second) -- handling the downstream injection risk. The consolidated discipline the team documents: treat LLM output as untrusted (it flows into sinks and can cause injection -- attacker-influenceable via prompt injection), handle it context-aware for each sink (HTML-encode for HTML display, parameterize for SQL, escape/avoid shell, sanitize markdown -- the sink's standard injection defense applied to the output), validate structured output (tool calls, schemas -- against the expected structure), handle markdown/link risks (XSS, data-leaking images, phishing) and downstream injection (output as input to other systems/LLMs), use schema enforcement (constraining the output shape), recognize the relationship to prompt injection (an injected prompt manifests as malicious output -- defend both sides), and monitor the output handling -- because LLM output flows into other systems and treating it as trusted creates injection vulnerabilities, so handling it as untrusted input to its sink (context-aware encoding/validation -- the sinks' standard defenses) is essential to secure LLM applications.