Network Isolation for Agents, in depth: zoning the agent stack, east-west NetworkPolicy, private model endpoints and a reachability test

What isolation has to stop

Think of each component as a possible starting point for an attacker and ask what it could reach next. Three starting points matter most in agent systems.

• The tool runner. It executes code or commands produced by a model, often from content an attacker wrote. Assume it is compromised on a bad day. If it shares a flat network with your databases, a single injected psql or curl reaches them.
• The orchestrator. It holds conversation state and calls tools and models. It is less exposed than the runner, but a confused-deputy bug can make it call something it should not.
• Retrieval and memory stores. Vector databases and caches are often deployed without authentication on the assumption that only internal services can reach them. On a flat network, that assumption is false.

The usual targets are equally predictable: the Kubernetes API server, the cloud instance metadata endpoint, unauthenticated internal admin interfaces, databases, and other tenants' workloads. Segmentation does not prevent the injection. It limits the blast radius, so a compromised runner can reach a proxy and nothing else.

Zone the stack by data flow

Start from data flow, not from teams or repositories. Group components by trust and by what they need to reach, then write the allowed edges as a short list. A typical agent stack has five zones.

Two rules make the model hold. Every edge is one-way: the orchestrator calls the tool runner, but the runner cannot open a connection back into Core. Results return over the same connection. And the Tools zone gets the strictest policy, because it runs the least trustworthy code. Put each zone in its own namespace, so policies can select by namespace, and do not let workloads in Tools set labels that grant entry to other zones.

NetworkPolicy semantics that cause most mistakes

Kubernetes NetworkPolicy is the usual mechanism for these rules. Its semantics are simple but different from a traditional firewall, and most misconfigurations come from five details.

1. It is only an allow-list. There are no deny rules. Policies that select the same pod are combined, and a connection is allowed if any of them allows it. You cannot override a broad allow with a narrow deny.
2. Isolation is per pod and per direction. A pod accepts all traffic until at least one policy selects it with Ingress in policyTypes. The same applies independently to Egress. A policy that lists only ingress leaves egress fully open.
3. Enforcement is done by the network plugin. The API server accepts policies even if the cluster's CNI ignores them. Check that yours enforces them, with a test, not by reading documentation.
4. Selectors combine in ways that surprise people. A namespaceSelector and a podSelector in the same from entry mean AND: pods with that label in that namespace. Written as two list entries, they mean OR: any pod in that namespace, or any pod with that label in the policy's own namespace. One extra dash changes the meaning.
5. Default deny breaks DNS. Once egress is denied, pods cannot resolve names until you allow traffic to the cluster DNS on port 53, over both UDP and TCP.

NetworkPolicy works at layers 3 and 4: pods, namespaces, IP blocks and ports. It cannot express "only GET /v1/search" or "only api.example.com". DNS-name rules and HTTP-aware rules exist, but as features of specific CNIs or service meshes, not of the standard API.

Policies for the tool zone

The Tools zone in YAML: deny everything, then allow ingress from the orchestrator and egress to DNS and the proxy. Namespace labels use kubernetes.io/metadata.name, which Kubernetes sets automatically and workloads cannot spoof by editing their own pods.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny, namespace: agent-tools}
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: runner-allow, namespace: agent-tools}
spec:
  podSelector: {matchLabels: {app: tool-runner}}
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: agent-core}}
      podSelector: {matchLabels: {app: orchestrator}}      # same entry: AND
    ports: [{protocol: TCP, port: 8080}]
  egress:
  - to:
    - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}
      podSelector: {matchLabels: {k8s-app: kube-dns}}
    ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
  - to:
    - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: egress}}
      podSelector: {matchLabels: {app: egress-proxy}}
    ports: [{protocol: TCP, port: 3128}]

The Data zone mirrors this from the other side: default deny, then ingress to the vector database only from pods labelled app: retriever in Core. Writing both sides means a mistake on one side is not enough to open a path. Add three pod-level settings to the runner, because the network is not the only path to the control plane: automountServiceAccountToken: false, no hostNetwork (host-network pods bypass pod policies on many CNIs), and a container sandbox as described in Agent Sandboxing with Docker and gVisor.

Identity above layer 4

Layer-4 rules answer "which pod may connect" but not "which caller may do what". For the paths that remain open, add identity. A service mesh with mutual TLS gives each workload a cryptographic identity derived from its service account, and its authorization policies can then allow, for example, only the retriever identity to call the vector database's query endpoint, and nobody to call its admin endpoint. Treat this as a second layer, not a replacement: if the mesh sidecar is bypassed or misconfigured, NetworkPolicy still holds, and the reverse is also true.

Self-hosted model servers deserve the same treatment. Many inference servers expose management or metrics endpoints alongside the generation API. Only the model gateway should reach them, and only the gateway should be reachable from Core.

Below the cluster: subnets and private endpoints

Kubernetes policy stops at the cluster boundary. Below it, use the cloud's own controls. Put node pools for the Tools zone in their own subnet with security groups or firewall rules that deny traffic to database subnets, so a container escape to the node still meets a boundary. Reach managed model APIs through private connectivity (AWS PrivateLink, Azure Private Link or Google Private Service Connect, depending on the provider and the service; check which endpoints your model service supports) so prompts and responses do not cross the public internet and the API can be restricted to your network. Block the instance metadata address from pods in every zone; the egress article covers the details. Turn on flow logs at this layer: they are often the only record of what a compromised pod tried to reach.

Worked example: an injected network scan

A user asks a research agent to summarise a web page. The page contains hidden text telling the agent to run Python that scans 10.0.0.0/16 for open ports, dumps any Redis it finds, and posts the result to an outside address. The model complies and the code runs in a tool runner. Here is what happens at each step.

1. The scan sends SYN packets to internal addresses. The runner's egress allows only DNS and the proxy, so every probe is dropped by the CNI. The scan reports nothing open, not even the vector database, because the Data zone also denies ingress from Tools.
2. The code tries the Kubernetes API at kubernetes.default.svc. Egress denies it, and there is no mounted token to use anyway.
3. It tries the metadata address. Denied by policy and by the node-level rule.
4. It posts to the outside address through the proxy. The proxy's allow-list does not include that domain, so the request fails and is logged with the pod's identity.
5. Policy-denied counters and proxy denials fire an alert. The session is flagged and the runner pod is replaced.

On a flat network, step 1 alone would have found the vector database and the cache, and steps 2 to 4 would have depended on luck. The data exfiltration article covers the channels that remain when some egress must stay open.

Proving it with a reachability matrix

Policies are code that nobody reads twice, so test them like code. Write the intended allow-list as a reachability matrix, run a probe from a pod in each zone, and fail the build on any difference. One subtlety: a refused connection means the packet reached a host that answered, so the network path is open. Only a timeout counts as blocked.

import socket, sys

EXPECT = {   # zone -> {target: should_connect}
    "tools": {"orchestrator.agent-core:8443": False, "qdrant.agent-data:6333": False,
              "postgres.agent-data:5432": False, "kubernetes.default.svc:443": False,
              "169.254.169.254:80": False, "egress-proxy.egress:3128": True},
    "core":  {"tool-runner.agent-tools:8080": True, "qdrant.agent-data:6333": False,
              "169.254.169.254:80": False},
    "retriever": {"qdrant.agent-data:6333": True, "postgres.agent-data:5432": False},
}

def path_open(target, timeout=3.0):
    host, port = target.rsplit(":", 1)
    try:
        socket.create_connection((host, int(port)), timeout=timeout).close()
        return True
    except ConnectionRefusedError:
        return True            # a host answered: the network path is open
    except OSError:
        return False           # timeout, unreachable or name not resolvable

zone = sys.argv[1]
bad = [(t, want) for t, want in EXPECT[zone].items() if path_open(t) != want]
for t, want in bad:
    print(f"{zone} -> {t}: expected {'open' if want else 'blocked'}")
sys.exit(1 if bad else 0)

Run it with kubectl exec from a probe pod carrying each zone's labels after every policy change and on a schedule, because CNI upgrades and new namespaces change behaviour. A test that has never failed proves nothing, so check once that it fails when you delete a policy.

Failure modes

• Ingress-only policies. Egress stays open because no policy lists it. Always set both policy types for exposed zones.
• A CNI that does not enforce. Policies apply cleanly and do nothing. Only the reachability test catches it.
• The OR selector. A stray dash admits every pod in a namespace. Review selectors in diffs and test from an unlabelled pod.
• Shared namespaces. Tool runners placed next to the orchestrator inherit its allow-list. Give each zone a namespace.
• Allow-all egress for convenience. A runner that "needs the internet" gets 0.0.0.0/0, which includes every private range. Send it through the proxy instead.

Trade-offs

Finer zones mean more policies to keep correct and more friction when a new tool needs a new path; five zones is a reasonable start, and per-tool namespaces are worth it only for tools with very different reach. L4 policy is cheap and widely supported but blind to identity and URLs; a mesh adds both at the cost of operating it. Per-session runner pods give the strongest isolation but add start-up latency; pooled runners are faster but must be reset between sessions. Whatever you choose, the reachability test is what turns the design into a guarantee. For multi-tenant platforms, combine this with the tenant boundaries in Tenant Isolation for LLM Systems.

What to do next

1. Draw your agent stack's actual data flows and group components into zones; write the allowed edges as a table.
2. Give each zone a namespace and apply default deny for both ingress and egress, starting with the tool runners.
3. Add explicit allows, including DNS, writing both the caller's egress and the callee's ingress.
4. Disable service account token mounting and host networking for tool runners, and block the metadata address.
5. Route managed model API traffic through private endpoints where your provider supports them.
6. Commit the reachability matrix and probe script, run it in CI and on a schedule, and prove it fails when a policy is removed.
7. Alert on policy-denied counters and proxy denials from the Tools zone.