Core concept
Inf2: up to 12 Inferentia2 chips. Runs 70B models. Serves via SageMaker or direct EC2. Neuron SDK.
Advertisement
How it works
Same Neuron ecosystem constraints as Trainium. Model must be supported. Some feature parity gaps with CUDA.
Advertisement
Trade-offs + gotchas
Production LLM serving at scale. Cost-sensitive inference. AWS-committed. Complements Trainium.