Why architecture matters here
Flink architecture matters because streaming is unusually correctness-sensitive. Watermarks, state, and checkpointing must all cooperate for the results to be right. Naive implementations either lose data on restart, miscount under lateness, or explode memory as state grows.
Cost matters. Flink clusters run continuously; state storage grows; checkpointing consumes IO. Right-sized clusters and state backend choices materially affect spend.
Reliability is what Flink delivers when configured correctly. Exactly-once with Kafka + Flink + idempotent sink is well-proven. Checkpointing survives arbitrary node failures. But every guarantee has a configuration prerequisite.
The architecture: every piece explained
Walk the diagram top to bottom.
Job. Your streaming application. DataStream API (imperative), Table API (relational), or Flink SQL. Compiled to a dataflow graph.
JobManager. The coordinator. Schedules tasks, coordinates checkpoints, handles failures. HA mode uses ZooKeeper for failover between JobManagers.
TaskManagers. Worker processes. Each has slots (default = CPU count); each slot runs one parallel subtask. Slots share the JVM to reduce serialization overhead.
Dataflow Graph. Directed graph of operators (source, filter, map, keyby, window, sink). Each operator has a parallelism; multiple parallel subtasks run per operator.
State Backends. RocksDB (durable, disk-based, large state), HashMap (heap, small state, fast). Snapshot to distributed filesystem (HDFS, S3) for checkpoints.
Sources + Sinks. Kafka is the most common source and sink. Flink also supports Kinesis, JDBC, file systems, and more. Custom sources implement a well-defined interface.
Windows. Tumbling (non-overlapping fixed intervals), Sliding (overlapping), Session (activity-based). Windows are the primary aggregation primitive.
Watermarks. Event-time progress markers. A watermark of time T means "no more events with event_time < T will arrive." Enables late-data handling and window closure.
Checkpointing. Periodic distributed snapshots of state. On failure, restore from last checkpoint. Chandy-Lamport algorithm.
Exactly-Once. Flink internal exactly-once + two-phase commit with sinks. Kafka source + Kafka sink is exactly-once out of the box in modern Flink.
End-to-end streaming job flow
Trace a job. You want to compute per-user-per-hour purchase totals from a Kafka stream of purchases.
Submit the job. JobManager parses the dataflow: KafkaSource → keyBy(user_id) → window(1 hour tumbling) → aggregate(sum) → KafkaSink.
JobManager schedules subtasks onto TaskManager slots. If parallelism = 8, the pipeline runs 8 concurrent instances of each operator.
KafkaSource reads from partitions and emits events. Watermarks flow along with events; the source generates watermarks based on event time.
keyBy hashes user_id to route each event to a specific downstream subtask. State (window aggregates) partitioned by user_id.
Windows collect events until the watermark passes the window's end. Then the aggregate computes and emits.
Every 60 seconds, checkpoint fires. Each operator flushes its state to S3. JobManager coordinates via Chandy-Lamport snapshots.
TaskManager crashes. JobManager notices via missed heartbeat. Restarts the failed subtasks on other TaskManagers. Restores state from last checkpoint. Rewinds Kafka offsets to match. Job resumes; no data lost.