Why architecture matters here
Flink state fails on operational subtleties. Growing state without TTL blows disks. Checkpoint duration exceeds interval and jobs stall. State schema evolution breaks restore. Rescaling requires savepoints with proper key groups.
The architecture matters because each of these has an architectural answer: TTL + cleanup, backend tuning, savepoint + schema versioning, rescale via savepoint.
Get the pieces right and Flink runs for months without touching operator intervention.
The architecture: every piece explained
The top strip is the state types. Operator is the user function processing events. Keyed state is per-key partition state — ValueState, ListState, MapState. Operator state is per-instance state (used for source offsets). State backend is HashMap (in-memory, small state) or RocksDB (disk-backed, large state).
The middle row is checkpointing. Checkpoints are async periodic snapshots stored to durable storage. Savepoints are manual, versioned snapshots suitable for upgrade or migration. Barrier alignment injects markers into streams so all operators snapshot a consistent cut. Restore + rescale reloads state and can change parallelism.
The lower rows are hygiene. TTL + cleanup bounds state growth. Metrics track state size and checkpoint duration. Ops handles schema evolution (typically via savepoint + updated code), backend tuning (block cache, compaction), and retention.
End-to-end flow
End-to-end: a Flink job maintains per-user session windows via keyed ValueState in RocksDB. Every 60s, checkpoint barrier propagates through operators; each snapshots its state to S3; job continues. State grows to 200 GB; incremental checkpoints keep durations under 30s. A code change is deployed via savepoint: stop with savepoint, upgrade code (schema compatible), resume from savepoint. Later a rescale: savepoint at 100 tasks, restart at 200 with same savepoint; Flink redistributes key groups. Metrics confirm no lag added.