Why architecture matters here
Pipelines architecture matters because the abstraction hides trade-offs you need to make explicit for production. A pipeline call is convenient; a pipeline call with the wrong device, wrong precision, or wrong batch size is slow or expensive. Knowing what the abstraction hides lets you configure it well.
Cost matters. Auto-selected defaults may pick a large model when a smaller one suffices. Batch size defaults may leave GPU capacity unused. Explicit configuration converts convenience into efficiency.
Reliability comes from understanding failure modes: OOM on large inputs, tokenizer mismatches, silent CPU fallback. All are common; all have specific fixes.
The architecture: every layer explained
Walk the diagram top to bottom.
User Code. pipeline("text-generation") or pipeline("automatic-speech-recognition", model="openai/whisper-large-v3"). One line.
Task Registry. Maps task name to a pipeline subclass. text-generation → TextGenerationPipeline; ASR → AutomaticSpeechRecognitionPipeline. Each has task-appropriate defaults and preprocessing.
Model Selection. If not specified, uses the task's default model from the registry. Specify explicitly with model="..." for control.
Preprocessor. Task-specific. Text tasks use the tokenizer; audio tasks use a feature extractor (spectrogram, mel filterbanks); image tasks use an image processor. Auto-loaded from model repo.
Forward Pass. Runs the actual model. For generation, calls model.generate() with default generation config; for classification, calls forward() and applies softmax.
Postprocessor. Decodes token IDs back to text; applies thresholds for classification; formats output as list of dicts.
Batching. Pipeline accepts a list of inputs; auto-batches them for GPU efficiency. Batch size configurable.
Device Placement. device=0 for GPU 0, device="cuda" for auto, device="cpu" for CPU. Auto-placement via device_map for large models.
Streaming Support. Some pipelines (text-generation) support streaming via TextStreamer or TextIteratorStreamer for token-by-token output.
Optimum Backends. Optimum wraps TensorRT-LLM, ONNX Runtime, Intel Neural Compressor. Same pipeline API, faster backend.
End-to-end pipeline call
Trace a call. You write: gen = pipeline("text-generation", model="meta-llama/Llama-3-8B-Instruct", device=0, torch_dtype=torch.bfloat16). Then gen("Explain quantum entanglement briefly.", max_new_tokens=200, temperature=0.7).
Pipeline constructor loads the model with the specified dtype onto GPU 0. Auto-loads tokenizer and generation config. Warmup with a dummy input.
On call, preprocessor tokenizes: "Explain quantum entanglement briefly." → list of token IDs. Adds special tokens.
Forward pass invokes model.generate() with the provided kwargs. Model runs on GPU; produces up to 200 tokens.
Postprocessor decodes token IDs back to text. Skips input tokens (default) so only new text returned. Formats as [{"generated_text": "Quantum entanglement is..."}].
Same call but batched: gen(["prompt1", "prompt2", "prompt3"], batch_size=3). Pipeline pads to the longest, runs one batched forward, splits results.
Streaming: streamer = TextIteratorStreamer(tokenizer). gen("prompt", streamer=streamer). Iterate over streamer to yield tokens as they generate. Perfect for chat UI.