Core concept
Supports major architectures. AWQ + GPTQ quantization. LoRA. Speculative decoding.
Advertisement
How it works
NVIDIA-first (recent AMD + TPU support). Fast-moving. Feature-rich.
Advertisement
Trade-offs + gotchas
Community LLM serving. Cost-conscious inference. Alternative to TRT-LLM.