Why architecture matters here
Accelerate architecture matters because distributed training is fiddly. Different backends have different APIs; wrong config wastes GPUs; getting FSDP + gradient checkpointing right is subtle. Accelerate wraps this into a coherent API.
Cost is proportional to hardware. Accelerate itself is thin overhead.
Reliability comes from tested integrations. Accelerate handles the annoying details; you focus on training logic.
The architecture: every backend explained
Walk the diagram top to bottom.
Training script. Standard PyTorch loop; add Accelerator wrapper.
Accelerator(). One object handles device placement, distributed setup, mixed precision, gradient accumulation.
Config. `accelerate config` interactive setup; produces YAML.
DDP mode. DistributedDataParallel; standard multi-GPU.
FSDP integration. Fully-Sharded Data Parallel; params/grads/optimizer states sharded across GPUs.
DeepSpeed integration. ZeRO stages 1/2/3 via Accelerate config.
Megatron-LM plugin. Tensor + Pipeline parallelism for very large models.
TPU / MPS support. Same script runs on TPU with XLA; MPS on Apple.
Mixed precision. bf16, fp16, fp8 auto-cast.
Save + resume. accelerator.save_state and .load_state for checkpointing.
End-to-end training script flow
Trace a script. Basic training loop: ``` accelerator = Accelerator() model, optim, dl = accelerator.prepare(model, optim, dataloader) for batch in dl: with accelerator.autocast(): loss = model(batch).loss accelerator.backward(loss) optim.step() ```
Run on single GPU: `python train.py`. Accelerator uses that GPU. Works.
Run on 4 GPUs: `accelerate config` then `accelerate launch train.py`. Config specifies DDP. 4 processes launched; each on one GPU; DDP wraps model.
Scale to FSDP: reconfigure to FSDP; same script. Weights + grads + optim states sharded.
Scale to DeepSpeed ZeRO-3: config specifies DeepSpeed; JSON with stage 3. Runs.
Massive model with Megatron: config with Megatron plugin. TP + PP handled.
Checkpoint: `accelerator.save_state(dir)`. Resume: `accelerator.load_state(dir)`. Works across configs (mostly).