Why architecture matters here
Watermark bugs are hard to spot. Data appears to be counted correctly until a late-arriving event 20 minutes past the window boundary is silently dropped. The architecture matters because event-time semantics require deliberate delay + output mode choices.
With the pieces mapped, you can define correctness for your streaming pipeline explicitly.
The architecture: every piece explained
The top strip is the semantics. Source reads events. Event time column annotates each row. Watermark is (max event time seen) − allowed delay; state older than watermark is safe to close. State store holds windowed aggregations.
The middle row is control. Trigger defines the processing-time cadence (continuous, once, or interval). Output mode — append (only closed windows), update (any changed row), complete (whole state) — must match sink. Late data policy decides whether to drop or route past-watermark data to a side output. Checkpoint to durable storage enables restart.
The lower rows are sinks + ops. Sink: Delta, Kafka, or files. Observability tracks batch duration + lag. Ops covers schema evolution, state pruning, and backfill.
End-to-end flow
End-to-end: job aggregates page views per 5-minute window with watermark delay 10 minutes. Trigger interval 1 minute. State grows during the last 10 minutes; watermark closes older windows and outputs final counts in append mode. A late event 8 minutes past its window still gets counted (inside the delay). A late event 12 minutes past is dropped (or routed via late-data sink). Checkpoints allow restart from last processed offsets. Observability shows batch duration steady + lag under 1 minute.