Why it matters

Ray Serve is popular Python serving. Understanding shapes options.

Advertisement

The architecture

Ray cluster: head + workers.

Deployments: replicas of models.

Router: routes requests.

Ray Serve stackRay clusterdistributedDeploymentsmodel replicasRouterload balanceRay Serve + vLLM common combo; KubeRay for K8s deployment
Ray Serve.
Advertisement

How it works end to end

Deployments: replicas + resources.

Autoscaling.

Batching per deployment.

KubeRay integration for K8s.