gRPC Load Balancing on Kubernetes: Why One HTTP/2 Connection Pins Traffic, and the Fixes

Balancing connections is not balancing requests

A Kubernetes ClusterIP Service is load-balanced at layer 4. kube-proxy, or an eBPF replacement, picks a backend Pod when a TCP connection is opened, and every packet on that connection goes to the same Pod. For HTTP/1.1 clients that open many short connections this spreads load well. gRPC runs on HTTP/2, which multiplexes many concurrent requests as streams over one long-lived connection. A gRPC channel with the default policy keeps one connection open for as long as it can. Every request from that client then goes to one Pod.

The symptom is familiar: a service with 12 replicas where 3 run hot and the rest idle, and scaling out changes nothing because existing clients never reconnect. DNS in microservices shows the short fix; this article explains the mechanics, the options and how each one fails.

The arithmetic of pinning

If C client connections are placed independently and uniformly at random on N Pods, the expected number of Pods that receive no connection at all is N(1 - 1/N)^C. With 4 client Pods, each holding one channel, and 12 server Pods, that is 12 x (11/12)⁴, about 8.5 Pods idle on average. Even with 24 connections, about 1.5 Pods stay idle on average and the busiest Pod usually carries two to three times the mean. The fix is not more replicas; it is getting requests, not connections, distributed.

Illustrative python, checked for syntax; adapt names and storage to your system.

def expected_idle(pods: int, connections: int) -> float:
    return pods * (1 - 1 / pods) ** connections

for conns in (4, 12, 24, 48):
    print(conns, round(expected_idle(12, conns), 2))

Option 1: client-side round_robin over every Pod

gRPC clients choose a load-balancing policy. The default, pick_first, connects to one address. round_robin connects to every address the resolver returns and spreads requests across the ready ones. To give the resolver every Pod address, point the client at a headless Service (clusterIP: None), whose DNS name returns the ready Pod IPs.

Illustrative python, checked for syntax; adapt names and storage to your system.

import grpc

SERVICE_CONFIG = '{"loadBalancingConfig": [{"round_robin": {}}]}'

channel = grpc.insecure_channel(
    "dns:///orders-headless.shop.svc.cluster.local:50051",
    options=[("grpc.service_config", SERVICE_CONFIG),
             ("grpc.keepalive_time_ms", 300_000)],
)

This distributes requests evenly across the Pods the client knows about. Its weakness is discovery: gRPC's DNS resolver re-resolves mainly when connections fail or are closed, not on the DNS TTL, so new Pods can stay unused for a long time after a scale-up. Round robin is also blind to load: a slow Pod gets the same share as a fast one. gRPC's weighted_round_robin policy uses load reported by the servers to correct that, at the cost of servers reporting it.

Watch the connection count as fleets grow. Every client holds a connection to every Pod, so 200 client Pods and 300 server Pods mean 60,000 HTTP/2 connections, each with keepalive traffic and TLS state. At that scale give each client a stable subset of servers, or move balancing into a proxy tier.

Option 2: MAX_CONNECTION_AGE on the server

gRPC's server-side connection management proposal (gRFC A9) defines MAX_CONNECTION_AGE: when a connection reaches that age, the server sends GOAWAY, lets outstanding RPCs finish within MAX_CONNECTION_AGE_GRACE, and closes it. A random jitter of plus or minus 10 per cent is added to spread reconnections. Clients reconnect, which goes back through the Service's layer-4 choice, and with a headless Service and round_robin it also triggers re-resolution, so new Pods are discovered. Both settings default to infinite.

Illustrative grpc-go server options; the Python and C++ equivalents are the grpc.max_connection_age_ms and grpc.max_connection_age_grace_ms channel arguments.

server := grpc.NewServer(grpc.KeepaliveParams(keepalive.ServerParameters{
	MaxConnectionAge:      5 * time.Minute,
	MaxConnectionAgeGrace: 30 * time.Second,
}))

Alone, behind a ClusterIP Service, this turns permanent pinning into periodic reshuffling. With 4 connections and 12 Pods the expected idle count is still about 8.5 at any instant; only which Pods are idle changes. It is a complement to round_robin, not a substitute. Set the grace longer than your longest normal RPC, and longer still for streaming RPCs, or they are cut every few minutes.

Option 3: an L7 proxy or mesh

A proxy that terminates HTTP/2, such as Envoy in a mesh sidecar, a node-level proxy, or a Gateway API implementation with GRPCRoute, balances each request rather than each connection. It watches EndpointSlices directly, so it sees new Pods without DNS, and it can use least-request or outlier detection. The cost is a hop, proxy CPU, and another component to operate and upgrade. It is the natural choice when you cannot change clients, or clients are in many languages.

Option 4: proxyless xDS

gRPC libraries can consume xDS, the same control-plane API Envoy uses, and apply the resulting load-balancing configuration in the client. This gives request-level balancing and fast endpoint updates without a sidecar, but it needs an xDS control plane and gRPC versions with the features you depend on. Treat it as an architectural decision for a platform, not a quick fix for one service.

Choosing

Failure modes

• Keepalive too aggressive. grpc-go servers enforce a minimum interval between client pings, five minutes by default. Clients that ping more often receive GOAWAY with too_many_pings and reconnect in a loop. Match client keepalive to the server enforcement policy.
• Grace shorter than streams. MAX_CONNECTION_AGE_GRACE cuts long streams; clients must handle UNAVAILABLE and resume.
• Stale DNS answers. A headless answer can include Pods that have since terminated. Clients retry on another address and re-resolve, but expect a burst of UNAVAILABLE during rollouts unless the server drains with GOAWAY first, as in pod termination without dropped requests.
• Retries concentrating load. Transparent retries and hedging go to other Pods; with pinning they all hit the few busy ones.
• Health versus readiness. Round robin only skips connections that fail. A Pod that accepts connections but answers slowly keeps its share.

How to verify

1. Export requests per second per Pod and compute the coefficient of variation across Pods; an even spread is well under 0.2, pinning shows values near or above 1.
2. Scale the server Deployment up during a load test and record how long until new Pods carry their share.
3. Count connections per Pod from server metrics or socket tables before and after enabling MAX_CONNECTION_AGE.
4. Run a rollout and confirm GOAWAY-driven reconnects produce no client-visible errors.

Practice and related reading

Further reading:

• DNS in microservices
• Connection pooling architecture
• Kubernetes pod termination without dropped requests

Sources and scope

Checked against these sources on 17 October 2026. Examples and figures are illustrative, not measurements.

• gRFC A9: server-side connection management
• grpc-go keepalive package
• gRPC service config documentation
• gRPC guide: custom load balancing policies
• gRPC blog: gRPC load balancing