Why it matters

Scale-out shapes high-QPS agent apps. Understanding enables real deployments.

Advertisement

The architecture

Stateless: instances hold no state; SessionService handles.

LB: distribute requests across instances.

Scale-out architectureStateless instancesno local stateLoad balancerdistribute requestsShared session + memoryexternal storesStateless instances enable easy horizontal scaling and rolling deploys
Scaling model.
Advertisement

How it works end to end

Load balancer: L4 (TCP) or L7 (HTTP) depending on transport.

Session affinity: not required if session state is external.

Autoscaling: based on request rate or CPU.

Rate limiting: per user or per instance.