Why architecture matters here

Streaming windows without watermarks either never fire or fire too early. Architecture matters because time + lateness + trigger compose.

Advertisement

The architecture: every piece explained

The top strip is time. Event time. Ingestion time. Watermark. Trigger.

The middle row is late handling. Late data. Allowed lateness. Side output. Retract + emit.

The lower rows are ops. Heuristic watermark. Metrics. Ops — source config + tuning.

Streaming watermarks — event time + late data + triggeradvance progress in unbounded streamsEvent timewhen it happenedIngestion timewhen we saw itWatermarkno earlier eventsTriggerfire windowsLate datapast watermarkAllowed latenessgrace windowSide outputvery late recordsRetract + emitcorrectionsHeuristic watermarkquantile of arrivalsMetricswatermark lag + latenessOps — source config + tuning + backfillhandlegraceroutecorrectestimatewatchwatchoperateoperate
Streaming watermark advances event-time progress.
Advertisement

End-to-end flow

End-to-end: source emits records with event time. Watermark advances as 99th-percentile of arrivals. When watermark passes window end, window fires. Late record within allowed lateness triggers retract + re-emit. Very late goes to side output.