Why architecture matters here
Service mesh architecture matters because it centralizes cross-cutting concerns that would otherwise scatter across every service. mTLS handled uniformly means every internal call is encrypted without each service reimplementing TLS. Retry policy enforced by the mesh means one team can't take down a shared backend with an aggressive retry storm. Traffic splitting for canaries is one config change instead of one deploy per service.
Cost is real. Each pod runs a sidecar proxy that adds 10-40 MB of memory and 0.1-0.3 vCPU. In a cluster with 10,000 pods, that's a meaningful bill. Ambient mesh (Istio) and node-local proxies (Cilium) address the overhead by moving proxying out of the pod, at the cost of some feature loss. Choose based on scale and workload.
Reliability requires care. The mesh becomes a critical dependency — control plane down means proxies degrade in specific, sometimes surprising ways. Istio proxies keep working with the last known config for a while but new pods can't join. Understanding the failure modes shapes how you deploy the mesh itself.
The architecture: every piece explained
Walk the diagram top to bottom.
Service pod. Application container plus a sidecar proxy in the same pod. The application talks only to localhost; the sidecar handles egress and ingress on its behalf. Application code doesn't know the mesh exists.
Sidecar Proxy. Envoy (or linkerd-proxy) intercepts all traffic. It does mTLS termination and origination, retries, timeouts, circuit breaking, load balancing, and telemetry. Configuration comes from the control plane via xDS.
Control Plane. Istiod (Istio) or linkerd-destination + linkerd-identity. It compiles high-level policies into Envoy configuration, distributes them to sidecars, issues certificates, and coordinates cluster-wide behavior. Multi-cluster mesh has one control plane per cluster with federation between them.
Traffic Policy. Virtual services (Istio) or ServiceProfiles (Linkerd) express routing rules: percent-based traffic split, header-based routing, retry budgets, timeout overrides. Applied uniformly to all traffic through the mesh.
Certificate Authority. Every workload has a SPIFFE ID. The mesh CA issues short-lived certificates (usually 24-hour lifetime) mapped to those IDs. Rotation is automatic. Compromise of one cert is contained by the short lifetime.
Config Distribution (xDS). Envoy's xDS API is how the control plane pushes routing, cluster, endpoint, and secret updates. gRPC streams keep sidecars in sync. Latency from policy change to enforcement is typically seconds.
Policy Store. RBAC rules ("service A can call service B on /api"), authentication policies, rate limits. Applied at the sidecar. Denies happen before the application even sees the request.
Telemetry pipeline. Every request generates L7 metrics (RPS, latency, error rate) tagged with source, destination, response code, path. Traces propagate through W3C headers. Metrics flow to Prometheus; traces to Tempo/Jaeger.
Gateway. An edge Envoy that handles north-south traffic — external clients coming in, or internal services calling external APIs. Gateways enforce the mesh's policies at the boundary.
End-to-end call flow
Trace a call. Service A wants to call service B on /api/order. The application does an HTTP GET to service-b.default.svc.cluster.local:8080. The kernel routes to localhost; the sidecar intercepts.
The sidecar checks the traffic policy: canary weights 5% to v2, 95% to v1. It picks v1 by weighted random. It applies retry policy (2 retries max), timeout (5 seconds), and circuit breaker (open if pool has 5 concurrent failures).
The sidecar looks up an endpoint for service-b-v1 from its endpoint discovery cache. It initiates an mTLS handshake with the target sidecar using its client certificate. The target sidecar validates the client's SPIFFE ID against RBAC policy: "service-a can call service-b on GET /api/*" — allow.
The target sidecar forwards the plain HTTP request to service-b on localhost:8080. Service B handles the request and responds. The response flows back through mTLS to service-a's sidecar and to the application.
Every hop generates metrics: request count, latency histogram, response code. Traces stitch together via W3C headers. If service-b times out, service-a's sidecar retries once. If retries exhaust, the circuit breaker opens and future requests fail fast for a cool-down period.
All of this — mTLS, retry, timeout, breaker, metrics, tracing — is invisible to both applications.