'Exactly-once' is one of the most confusing terms in streaming because it means different things to different systems. It is not a guarantee against duplicate processing in the user's code — it is a guarantee that the output reflects each input exactly once even after failures.
The three delivery semantics
At-most-once: messages may be lost, never duplicated. At-least-once: messages may be duplicated, never lost. Exactly-once: each message contributes exactly once to the final state. The third is what users want; the first two are what you get cheaply.
How Kafka does it
Kafka exactly-once requires three things: (1) idempotent producer — broker dedupes messages with same producer-id + sequence-number; (2) transactional producer — group writes across partitions into a two-phase commit; (3) read_committed consumer isolation level. With all three, end-to-end exactly-once across topics works.
How Flink does it
Flink uses asynchronous distributed checkpointing (Chandy-Lamport). Every N seconds, all operators snapshot their state and align on a checkpoint barrier. On failure, the entire job rewinds to the last consistent checkpoint. Combined with transactional sinks (Kafka, JDBC), this gives end-to-end exactly-once.
The fine print
Exactly-once is guaranteed only within the streaming system's boundary. If your job writes to an external service (e.g., a REST API) without idempotency keys, you can still double-write on failure. Always design sinks to be idempotent — exactly-once at the framework level is necessary but not sufficient.
Cost of exactly-once
Transactions add 5-15% latency overhead in Kafka. Checkpoint frequency in Flink directly trades throughput for recovery RPO. If your data is naturally idempotent (e.g., state-replacement upserts), you can usually achieve effectively-once with at-least-once delivery + idempotent processing — often simpler and faster than full transactional exactly-once.