SOC 2 is often the first security question an enterprise buyer asks, and adding a language model to a product changes the answer in ways that are easy to miss. Prompts now carry customer data to a third party. Retrieval indexes hold copies of documents. Logs contain free-text conversations. Behaviour changes when someone edits a prompt or a provider updates a model, without any application code changing. None of this is named in the SOC 2 criteria, which predate these systems and do not mention language models. Mapping them onto an LLM feature is your job.
This article explains what a SOC 2 report is, how to draw the system boundary around an LLM feature, how to handle the model provider in the report, how the criteria translate into concrete controls over prompts, models, retrieval and logs, and how to produce evidence an auditor can sample. It ends with a worked readiness plan and a checklist. It is engineering guidance, not legal or audit advice; your auditor decides what is sufficient.
What a SOC 2 report is, and what it is not
A SOC 2 report is an attestation by an independent CPA firm, performed under AICPA attestation standards, on a service organization's description of its system and on its controls, measured against the Trust Services Criteria. It is not a certification and there is no pass mark. The auditor issues an opinion, and the report lists each control, the test performed and any exceptions found. Customers read the exceptions.
A Type 1 report assesses whether controls are suitably designed at a point in time. A Type 2 report also tests whether they operated effectively across an observation period, commonly somewhere between three and twelve months, by sampling occurrences: a set of changes, a set of access reviews, a set of incidents. Enterprise buyers usually want Type 2.
The criteria are grouped into five categories. Security, whose points are called the common criteria and numbered CC1 to CC9, is always in scope. Availability, Confidentiality, Processing Integrity and Privacy are optional and chosen by the organization. For an LLM feature the choice of optional categories is the first real decision, and it is covered below.
Drawing the boundary
The system description defines what the report covers. For an LLM feature it has to name every component that stores or transforms customer data, including the ones that arrived with the model: the orchestration service that assembles prompts, the prompt templates and tool definitions, the vector store and its ingestion pipeline, conversation logs, evaluation datasets, and any fine-tuning data. A frequent gap is a retrieval index built from customer documents that lives in a separate account nobody listed.
Draw the data flow first, then the boundary. Each arrow that crosses the boundary is a question the auditor will ask: who is on the other side, what data crosses, and what control governs it.
The model provider is a subservice organization
A hosted model API that processes customer prompts is a subservice organization in SOC 2 terms: a vendor whose controls are relevant to your commitments. There are two ways to present it. The inclusive method brings the provider's controls into your report and requires the provider's cooperation with your auditor, which large providers rarely offer. The carve-out method excludes the provider's controls, names the provider in your description, and lists the complementary subservice organization controls, known as CSOCs, that you assume the provider operates, such as restricting access to customer prompts and honouring the agreed data retention.
Carve-out is the normal choice, and it moves the work into vendor management, which falls under CC9.2. Obtain the provider's own SOC 2 report, read its exceptions and its complementary user entity controls, and record that review every year. Record the contractual data terms that matter to you, such as whether prompts may be used for training and how long they are retained, from the provider's current agreement rather than from marketing pages, because those terms change. If a report's period ends months before yours, ask for a bridge letter covering the gap.
Your report in turn lists complementary user entity controls, or CUECs, for your customers: for example, that they configure single sign-on, that they do not submit data classes your terms exclude, and that a human reviews model output before it drives a consequential action. The guide to securing third-party LLM APIs covers the technical side of the provider boundary.
Mapping the criteria onto an LLM system
Most of the work is ordinary security engineering pointed at new objects. The table maps the criteria that change most when a model is added. Criterion numbers refer to the 2017 Trust Services Criteria, whose points of focus the AICPA revised in 2022.
| Criterion | What it asks | LLM-specific control |
|---|---|---|
| CC3 risk assessment | Identify and analyse risks to objectives | A threat model covering prompt injection, data leakage through outputs and tool misuse, reviewed yearly and on major change |
| CC6 logical access | Restrict access to data and systems | Provider keys in a secrets manager and rotated; retrieval filtered by tenant before the prompt is built |
| CC7 system operations | Detect and respond to anomalies and incidents | Monitoring for injection attempts, abuse and cost spikes, with LLM incident types in the response runbook |
| CC8 change management | Authorise, test and approve changes | Prompts, tool definitions, model versions and retrieval configuration versioned and deployed through the same gate as code |
| CC9.2 vendor risk | Assess and manage vendor risk | Annual review of the provider's report, terms and CSOCs |
| C1 confidentiality | Protect and dispose of confidential data | Retention limits and deletion for conversation logs, embeddings and eval sets built from production data |
| PI1 processing integrity | Processing is complete, valid, accurate | Only if in scope: output validation, eval thresholds and human review, stated precisely |
Access control is where LLM systems most often fail in practice, because retrieval can pull another tenant's document into a prompt without any access check failing. The enforcement patterns are in multi-tenant LLM isolation; for SOC 2, what matters is that the control exists, is tested and produces evidence.
Change management for things that are not code
An auditor testing CC8 will pick a sample of production changes and ask for the ticket, the test evidence and the approval for each. In an LLM system, the changes that most alter behaviour are often not code: a prompt edit, a new tool description, a switch to a newer model version, a change to retrieval settings. If those bypass the pipeline, they are unauthorised changes, and a Type 2 sample that finds one becomes an exception.
Treat all of them as deployable artifacts. Keep prompts and tool definitions in the repository. Pin the model to a dated version rather than an alias that the provider can move. Run an evaluation suite against the exact artifacts being deployed, and let a gate block the release unless the scores meet thresholds and someone other than the author approved it. The gate should also write a record of what it decided, because that record is the evidence.
# ci/release_gate.py: blocks a deploy and writes the evidence the auditor will sample
import hashlib, json, sys, datetime, pathlib
cfg = json.load(open("llm_release.json")) # model id, prompt files, eval run id
prompt_hash = hashlib.sha256(b"".join(
pathlib.Path(f).read_bytes() for f in sorted(cfg["prompt_files"]))).hexdigest()
evals = json.load(open(f"evals/{cfg['eval_run_id']}.json"))
problems = []
if cfg["model"].endswith("-latest"):
problems.append("model must be pinned to a dated version, not an alias")
if evals["prompt_hash"] != prompt_hash or evals["model"] != cfg["model"]:
problems.append("eval run does not match the artifacts being deployed")
for suite, score in evals["scores"].items():
if score < cfg["thresholds"][suite]:
problems.append(f"{suite} {score} below {cfg['thresholds'][suite]}")
if cfg["approved_by"] == cfg["author"]:
problems.append("approver must differ from author")
record = {"at": datetime.datetime.now(datetime.timezone.utc).isoformat(), "model": cfg["model"],
"prompt_hash": prompt_hash, "eval_run": cfg["eval_run_id"],
"author": cfg["author"], "approved_by": cfg["approved_by"],
"change_ticket": cfg["ticket"], "result": "blocked" if problems else "passed",
"problems": problems}
print(json.dumps(record)) # shipped to the append-only evidence log
sys.exit(1 if problems else 0)The evaluation suite itself is covered in LLM security evaluations. Emergency changes still need a documented path: allow a break-glass deploy, but have it open a ticket automatically and require retrospective approval within a stated time.
Evidence an auditor can sample
A Type 2 examination is a sampling exercise, so every control needs a population the auditor can draw from and an artifact for each item. Controls that exist only as policy text produce exceptions. Controls that emit records automatically are cheap to evidence.
| Control | Evidence artifact | Population sampled |
|---|---|---|
| Release gate | Gate record per deploy, linked to ticket and eval run | All production deploys in the period |
| Access review | Quarterly export of who can read logs, prompts and the vector store, with sign-off | Each quarter |
| Key rotation | Secrets manager rotation history for provider keys | All keys |
| Log retention | Lifecycle policy configuration plus a deletion job's run log | Job runs |
| Incident response | Tickets for LLM incidents with timeline and root cause | All incidents |
| Vendor review | Annual review memo of the provider's report and terms | Each in-scope vendor |
Conversation logs are both evidence and a liability. They must be complete enough to reconstruct an incident, and they contain customer data that the confidentiality criteria require you to protect and eventually delete. Design them once, deliberately, as described in LLM audit logging architecture, with redaction and a retention period you can state in the system description.
Processing Integrity: include it only if you can evidence it
Processing Integrity asks whether processing is complete, valid, accurate, timely and authorised. A language model's output is probabilistic, so a blanket claim of accuracy cannot be tested. Many organizations therefore leave this category out of scope for the LLM feature. If customers need it, scope it narrowly to what you can test: that every request receives a response or a recorded error, that structured output is validated against a schema before use, and that outputs below a confidence threshold are routed to a human. Those are claims an auditor can sample.
Worked example: the first Type 2 for a support copilot
A SaaS company adds a support copilot that drafts replies from each customer's help-centre articles, using a hosted model API. It already has a Type 2 report for its core product with Security and Availability in scope.
Month 1, a gap assessment finds four problems. Prompts are edited in an admin console outside the pipeline. The model is called by an alias. Conversation logs have no retention limit. The vendor inventory does not list the model provider. Month 2, the team moves prompts into the repository, pins the model version, adds the release gate, sets a 90-day retention on logs, adds the provider to the vendor register with a review memo, and adds Confidentiality to scope because customers ask about their help-centre data. Month 3, the auditor tests design as part of a Type 1 on the updated description. Months 4 to 9 are the observation period, during which the gate records, access reviews and deletion runs accumulate. The Type 2 report follows. Processing Integrity stays out of scope and the CUECs state that agents review every drafted reply before sending.
Failure modes
- Unlisted components. A vector store or eval bucket outside the described system holds customer data with no controls tested.
- Floating model versions. An alias changes behaviour mid-period with no change record.
- Out-of-band prompt edits. A console that edits production prompts is an unauthorised change path.
- Logs that outlive their purpose. Free-text conversations kept indefinitely contradict the confidentiality commitments in the description.
- Stale vendor terms. Retention and training terms recorded once and never rechecked.
- Overclaiming accuracy. Putting Processing Integrity in scope with controls nobody can sample.
- Unsanctioned AI use. Staff pasting customer data into personal AI accounts, outside every control. Cover it in the acceptable-use policy and in secrets and data handling training.
What to do next
- Draw the data flow for your LLM feature and list every store that holds customer data, including embeddings, logs and eval sets.
- Add the model provider to the vendor register, obtain its SOC 2 report and write down the CSOCs you rely on and the CUECs you pass to customers.
- Move prompts, tool definitions and model pins into the repository and put a release gate in front of them that writes evidence records.
- Set and enforce retention for conversation logs and derived data, and log each deletion run.
- Decide deliberately whether Confidentiality and Processing Integrity belong in scope, and scope any integrity claims to testable controls.
- Run an internal mock sample of ten changes and one quarter of access reviews before the observation period starts.