LLM Deployment Hardening

Container Security: Minimal Blast Radius

The container is your first perimeter. A container that can be broken into is a container that can leak secrets or model artifacts to an attacker. Start with distroless base images—python:3.11-slim still includes package managers and shells; distroless removes them. This means an attacker who achieves code execution inside the container cannot install tools, cannot read files outside their workload, and cannot easily pivot to other containers.

Run the container as a non-root user. Even if an attacker reaches arbitrary code execution, they do not immediately own the entire host. Set USER nobody or create a dedicated low-privilege user in the Dockerfile, and the OS will enforce that your application cannot read /root, cannot modify system binaries, and cannot write to system directories. On Kubernetes this is further enforced by the SecurityContext runAsNonRoot: true and allowPrivilegeEscalation: false.

Mount the filesystem as read-only where possible. If your LLM service reads the model from a volume, reads config from a ConfigMap, and only needs to write logs to stdout (not to a file), mount the root filesystem as read-only and request only a writable /tmp via emptyDir. An attacker who cannot write to the filesystem cannot persist malware or backdoors. Scan the resulting image with Trivy or Grype before pushing. These tools check for known CVEs in system packages and dependencies; a weekly re-scan of running images catches newly disclosed vulnerabilities before an attacker notices them.

Network Isolation: Deny by Default

Network segmentation is where you stop lateral movement. Assume any other service in your cluster is compromised and design egress so that your LLM deployment cannot reach it without explicit permission. Use Kubernetes NetworkPolicies (or Cilium) to allowlist only the egress destinations your workload actually needs: the LLM API provider (OpenAI, Anthropic, Bedrock), your database, your session store, your observability backend, and nothing else. A rule like to: [{cidr: '10.0.0.0/8'}] is a red flag—it is too broad and defeats the purpose of network hardening.

Enforce mutual TLS (mTLS) on all egress. Your deployment should never send API keys or session tokens over plaintext HTTP. Use https:// with certificate pinning if possible; if the provider supports it, pin their public key and reject any certificate chain that does not match. For internal services, use Istio or Linkerd to automatically inject mTLS sidecar proxies and verify every connection at the network layer.

Use private endpoints when the LLM provider offers them. AWS PrivateLink endpoints for Bedrock, Azure Private Link for Azure OpenAI, and similar mechanisms route traffic through AWS or Azure's internal network, never touching the public internet. This eliminates the risk that an attacker on the open internet can intercept or flood your API calls. If private endpoints are not available, use a VPN gateway or proxy as the egress chokepoint, inspect and log all outbound traffic, and ensure rate limits are enforced per destination.

Secrets Management: Keys Are Not Environment Variables

API keys and authentication credentials are the highest-value targets in an LLM deployment. Never store them in environment variables, never commit them to Git, and never log them. Use a secrets vault—HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager. On startup, your deployment fetches the secret from the vault using a workload identity (IRSA on AWS, Workload Identity on GCP, Managed Identity on Azure), not a static credential. The vault is the source of truth; the running service never holds a long-lived key.

Rotate secrets automatically every 30–90 days. When you store a secret in a vault that supports rotation, you configure a rotation policy and the vault handles the rest: it calls your provider's key rotation API, verifies the new key works, and then marks the old key as deprecated. Your application continues to fetch the latest key on each startup, so it never uses a stale credential. If a key is leaked or suspected compromised, rotate immediately—most providers support emergency rotation that takes seconds, and the vault updates the key without downtime.

Grant minimal privilege to each credential. If a deployment only needs to call Claude's Messages API, its API key should be scoped to that endpoint, not to Anthropic's entire API surface. If your LLM proxy only reads from a database, use a read-only database role tied to that proxy's credentials. In AWS, use IAM policies; in GCP, use IAM roles; in Azure, use RBAC. Document the minimum permissions required and audit quarterly to ensure no unnecessary access has accumulated.

Input Validation and Rate Limiting

Every prompt that reaches your LLM service should be validated before being passed to the model. Schema validation is the first step: check that the request has the expected shape, that required fields are present, and that strings are not longer than your safety policy allows. A prompt longer than 8,000 tokens might be exploitative (testing for jailbreaks or data leakage), so set a max length and reject requests that exceed it.

Content filtering on ingress catches abuse before it reaches the model. Check incoming prompts for signals of known jailbreak patterns (e.g., token smuggling, hypothetical role-play framing, DAN-style escalation) using regex or a lightweight classifier. This is not a silver bullet—it is a signal, not a guarantee—but it stops the most obvious attacks and keeps your API from being used as a harm generator without your knowledge.

Rate limiting per user, per IP, and per API key prevents abuse. A single IP or API key should not be able to send 10,000 requests per minute; a reasonable limit is 100–1000 per minute per key and 10,000–100,000 per hour per user (depending on your SLA). Use a distributed rate limiter (Redis, DynamoDB) so limits are enforced across all your replicas, not per-instance. When a user exceeds the limit, return a 429 Too Many Requests status and a Retry-After header. Log the event so you can spot patterns of attack.

Cost-based rate limiting is a modern variant: assign a cost to each request based on input tokens, output tokens, and model size, then track cumulative cost per user per billing period. This prevents a user from exhausting your budget by making many cheap requests or a few expensive ones, and it ties abuse prevention directly to business incentives.

Output Filtering and Content Moderation

Rate your model's responses before sending them to the user. Output filtering checks for two risks: (1) sensitive data leakage—did the model emit training data, PII, API keys, or credentials in its response? (2) harmful content—did the model produce instructions for violence, hateful speech, or other policy violations? Run every response through a content moderation API (OpenAI's Moderation endpoint, AWS Comprehend, Perspective API) and log the score. If the score exceeds a threshold, redact the response, log the incident, and return a generic failure message to the user.

Implement a data exfiltration detector to catch subtle leakage. Check for patterns that look like cryptographic keys (base64 + length), email addresses, phone numbers, credit card numbers (Luhn), or known secret prefixes (AWS key format, GitHub token format). A simple regex-based scanner catches obvious cases; for harder cases, a small fine-tuned classifier trained on your domain is worth the investment. Any detected exfiltration should trigger an alert and incident investigation.

Log all responses at the DEBUG level (not production logs—store them in a separate, access-controlled audit log). You will need the full response text if an incident or compliance investigation occurs. Encrypt logs at rest, retain them for 90 days by default, and make retrieval require multiple approvals to prevent casual snooping.

Monitoring and Alerting

If you cannot see it, you cannot defend it. Set up structured logging that captures every request and response as JSON, including the user ID, timestamp, prompt length, model name, output length, latency, cost, and any error messages. Ship these logs to a log aggregation platform (Datadog, Splunk, ELK, or Loki) so you can query and alert on patterns.

Set up real-time alerts for suspicious activity. Examples:

• More than N failed authentication attempts from a single IP in the last minute
• More than N rate-limit violations from a single user in an hour
• Average response time exceeds X seconds (possible DoS or resource exhaustion)
• More than M errors per minute (possible crash loop or deployment issue)
• Unusual spike in token consumption per user (possible abuse)
• Output filter triggered (possible model jailbreak or data leakage)

When an alert fires, route it to an on-call engineer. Do not ignore it; a single alert skipped today is ten tomorrow. Pair alerts with dashboards that show error rates, token costs, latency percentiles, rate-limit hits, and filter scores in real-time. During an incident, the dashboard is your control center.

Access Control and Audit Logging

Principle of least privilege means every human and service that touches your LLM deployment should have only the access they need. Role-based access control (RBAC) is the standard: define roles (engineer, on-call, security, compliance) and bind each person to one or more roles. Each role has minimal permissions. A junior engineer should not be able to modify production secrets; a security engineer should not be able to deploy code. Use your cloud provider's IAM system or a tool like Teleport to enforce this.

Require multi-factor authentication (MFA) for any access to production systems, even local login. If someone's laptop is stolen or their password is phished, the attacker still cannot access your deployment without the second factor.

Audit every action that touches production: API calls, deployments, secret access, configuration changes. Log the who, what, when, and why (include the ticket or justification). Immutable audit logs are a requirement for SOC2, ISO 27001, and most compliance frameworks. Store audit logs in a separate, access-controlled system, encrypt them, and retain them for 1–2 years minimum. Your cloud provider's CloudTrail, GCP Audit Logs, or Azure Activity Log will handle this automatically if configured.

DDoS Protection and Resource Limits

Distributed denial of service attacks aim to exhaust your resources and take the service offline. The naive defense—just buy more capacity—is expensive and does not work. Instead, layer protections. At the edge, use a DDoS mitigation service (Cloudflare, AWS Shield, Akamai) that absorbs volumetric attacks (floods of traffic) before they reach your infrastructure.

Inside your deployment, set hard resource limits on your containers and pods. On Kubernetes, set resources.limits.cpu and resources.limits.memory; the kubelet will kill any pod that exceeds these limits. This prevents a single misbehaving model or request from consuming all CPU or memory and crashing the entire node. Pair limits with horizontal pod autoscaling: if CPU usage exceeds 70%, spin up another replica automatically. This gives you capacity headroom without manual intervention.

Implement request timeouts to prevent slow-client attacks. If a client sends a request but never reads the response, do not let that connection hang forever. Set an HTTP timeout (30–60 seconds typical) and close the connection. If a model takes longer than your timeout, fail the request and log the event for investigation.

Model Versioning and Secure Updates

Do not deploy a new model version directly to production. Canary deployments route a small percentage of traffic (1–5%) to the new version and the rest to the stable version. Monitor error rates, latency, and output quality for the canary; if all metrics are healthy, gradually increase the traffic (10%, 25%, 50%, 100%) until the new version takes full load. If you detect a problem at any step, roll back instantly to the previous version.

Blue-green deployments are an alternative: run two identical production environments (blue and green), serve all traffic from one, and deploy the new version to the other. Once the new version is healthy, flip the load balancer to point to it. If anything goes wrong, flip back in seconds. Blue-green is safer than canary for models where even a tiny error rate is unacceptable.

Store all model versions in an artifact repository (Artifactory, Nexus, S3 with versioning, or a container registry) with immutable tags. Label each version with its build date, commit hash, and a GPG signature. Never delete a version unless you have a written policy allowing it. You may need to audit which version handled a request three years from now; the version history is your evidence trail.

Security patch your model and dependencies regularly. Subscribe to security mailing lists for Python, your ML framework, and any LLM SDKs you use. When a patch is released, test it in staging within 48 hours and deploy to production within 2 weeks for critical issues. A known unpatched vulnerability is a compliance violation and an exploit vector.

Compliance and Regulatory Requirements

Depending on your jurisdiction and industry, you may need to comply with regulations like GDPR (if serving EU users), HIPAA (if handling health data), PCI-DSS (if processing payment cards), SOC2 (if selling to enterprises), or others. Compliance is not a checkbox; it is a continuous process. Start by documenting your data flows: what data enters your system, where it goes, how long it is retained, and how it is deleted. Use a data flow diagram and keep it updated.

Implement data minimization: collect only the data you actually need. If you do not need the full user address, do not ask for it. If you do not need to retain prompts indefinitely, delete them after 30 days (or whatever your policy is). GDPR's right to erasure means if a user asks you to delete their data, you must do it (with exceptions for legal holds). Design your data schema and deletion procedures to make this easy.

Maintain incident response procedures and test them quarterly. If a data breach occurs (a model accidentally leaks training data, an API key is compromised, a server is hacked), you must notify affected users within 72 hours (GDPR) or 30 days (most US state laws). Have a playbook: who to notify, what to say, which channels to use, and how to investigate the root cause. Practice it at least once per year.

Keep an audit trail of all compliance actions: security training completed, vulnerability scans run, penetration tests conducted, incidents investigated. Your auditors will ask for evidence; if you have not documented it, it did not happen (from a compliance perspective).

Testing and Validation

Hardening practices are only as good as your testing. Set up automated security tests that run on every commit and every deployment:

• SAST (Static Application Security Testing): Scan your code for secrets (truffleHog, TruffleHog3), hardcoded credentials, and common vulnerabilities (Bandit for Python, similar tools for your language).
• Dependency scanning: Check your requirements.txt or package.json for known CVEs (Safety, Snyk, Dependabot).
• Container scanning: Scan the built Docker image for vulnerabilities (Trivy, Grype, Anchore). Fail the build if critical CVEs are found.
• DAST (Dynamic Application Security Testing): Run your deployment in staging and fuzz it—send malformed inputs, oversized payloads, SQL injection attempts, prompt injection attempts—and verify it handles them gracefully (returns errors, does not crash or leak secrets).
• Regression tests for security: If a security bug is found and fixed, write a test that would have caught it and add it to your suite. This prevents the same bug from being re-introduced.

Threat modeling is another practice worth formalizing. Bring together engineers, product, and security to map out the attack surface of your LLM deployment. What are the assets? Who are the threats? What is the impact of a breach? Where are the weaknesses? STRIDE or PASTA methodologies provide structure. The output is a risk register—a living document of known risks and the mitigations in place.

Penetration testing by an external firm annually or semi-annually is standard for any service that handles sensitive data or serves many users. A professional red team will find vulnerabilities your internal testing misses. Use their findings to drive security improvements and demonstrate due diligence to your auditors and users.