Why it matters

Preemption inevitable at scale. Understanding shapes reliability.

Advertisement

The architecture

Detect: signal handler.

Save state: checkpoint.

Resume: on new instance.

Preemption handling flowPreemption signalSIGTERM / APISave statecheckpointRestarton new instanceK8s Pod Disruption Budgets + spot signals coordinate graceful shutdown
Preemption.
Advertisement

How it works end to end

SIGTERM signal handler.

Async checkpoint mid-training.

K8s: preStop hook + PDB.

Recover from last checkpoint.