Why architecture matters here
Windowing matters because it's the fundamental technique for aggregating unbounded streams -- bounding the infinite into finite windows -- and it's inseparable from the hard problem of time semantics (event time, watermarks, late data) that makes stream processing correct. Streams are unbounded (never-ending), but aggregations need bounded data -- so windowing (dividing the stream into finite windows) is how you aggregate streams (counts, sums, averages over windows -- e.g., requests per minute, average over the last 5 minutes). This is fundamental to stream processing (most stream analytics are windowed aggregations). And windowing is inseparable from time semantics: correct windowing needs event time (windowing by when events happened -- since they arrive out of order/late), which requires watermarks (tracking event-time progress) and late-data handling -- the hard, crucial part of stream processing correctness. For stream processing (increasingly important -- real-time analytics, monitoring, event processing), windowing (and its time semantics) is a core concept, and understanding it (the window types, event time, watermarks, late data) is understanding how to aggregate streams correctly.
The window-types insight frames the aggregation semantics, and choosing the right type is key. The three window types serve different aggregation semantics. Tumbling (fixed, non-overlapping -- each event in exactly one window): for non-overlapping periodic aggregations (e.g., the count per each 1-minute period -- distinct minutes). Sliding (fixed-size, overlapping -- each event in multiple windows): for moving/rolling aggregations (e.g., the average over the last 5 minutes, updated every minute -- a rolling window -- so each event contributes to multiple overlapping windows). Session (activity-based, dynamic-sized -- a window per burst of activity, ending after an inactivity gap): for grouping related events (e.g., a user's session -- the events in a burst of activity, the session ending when the user is inactive for a gap -- dynamic, per activity). So the window type is chosen by the aggregation semantics: tumbling for distinct periodic aggregations, sliding for rolling/moving aggregations, session for activity-based grouping. Choosing the right type is key (matching the windowing to the desired aggregation semantics -- e.g., using tumbling when you need distinct periods, sliding when you need a rolling window, session when you need activity-based grouping). Understanding the window types (tumbling, sliding, session -- and their semantics) is understanding how to frame stream aggregations.
And the event-time-plus-watermarks reality is the hard core, and it's what makes stream windowing correct despite out-of-order and late data. The crucial complication: events don't arrive in order or immediately. An event that happened at time T (its event time) might arrive and be processed later (its processing time) -- and events can arrive out of order (a later-happening event arriving before an earlier-happening one) or late (long after they happened -- network delays, buffering). For correct windowing, you usually need event time (windowing events by when they happened -- so an event goes in the window for its event time, regardless of when it arrived) -- because processing-time windowing would put events in the wrong windows (by arrival, not occurrence -- incorrect for most analytics). But event-time windowing creates a problem: how do you know when a window is complete (all its events have arrived) so you can emit its result? Since events can be late, you can't just wait a fixed time. Watermarks solve this: a watermark tracks the progress of event time (asserting 'event time has advanced to X -- so events with event time before X have probably all arrived') -- so when the watermark passes a window's end, the window is considered complete (its result emitted). And allowed lateness handles late data (events arriving after the watermark -- either dropped, or accommodated by keeping the window state a bit longer -- the allowed lateness -- and updating the result). So event time (correct windowing by occurrence) plus watermarks (tracking event-time progress to know when windows are complete) plus late-data handling (allowed lateness) is the hard core of correct stream windowing (handling out-of-order and late events). Understanding the event-time-plus-watermarks reality (event-time windowing, watermarks for completeness, late-data handling) is understanding the hard core of correct stream windowing.
The architecture: every piece explained
Top row: the problem and window types. The problem: aggregating an unbounded (infinite) stream -- which needs bounded data (windows) to aggregate. Tumbling windows: fixed-size, non-overlapping (each event in exactly one window -- e.g., each 1-minute period) -- for distinct periodic aggregations. Sliding windows: fixed-size, overlapping (each event in multiple windows -- e.g., a 5-minute window every minute) -- for rolling/moving aggregations. Session windows: activity-based (a window per burst of activity, ending after an inactivity gap -- dynamic-sized) -- for activity-based grouping (e.g., user sessions).
Middle row: time and lateness. Event vs processing time: event time (when the event happened -- for correct windowing) vs processing time (when it's processed) -- the crucial distinction (correct windowing usually needs event time, since events arrive out of order/late). Watermarks: tracking event-time progress (a watermark asserting event time has advanced to X -- so windows before X are likely complete) -- to know when to emit windows despite out-of-order arrival. Late data: events arriving after the watermark (late) -- handled by allowed lateness (keeping window state longer to accommodate late events, updating results -- or dropping them). Triggers: controlling when window results are emitted (at the watermark -- window complete; or early/late triggers -- emitting partial results early, or updating on late data) -- the emission timing.
Bottom rows: state and joins. Window state: accumulating the aggregation per window (each window's running aggregation -- the state -- held until the window is emitted and cleaned up) -- the per-window state. Windowed joins: joining two streams over windows (matching events from two streams within a window -- e.g., joining orders and shipments within a time window) -- windowed stream joins. The ops strip: window sizing (choosing the window size -- balancing the aggregation granularity against the state and latency -- larger windows more state and latency, smaller more granular), state (managing the window state -- it grows with the windows and their data; bounding it -- cleaning up completed windows, the allowed lateness affecting how long state is kept), and lateness tuning (tuning the watermark and allowed lateness -- balancing completeness -- accommodating late data -- against latency -- waiting longer -- and state -- keeping windows longer).
End-to-end flow
Trace an event-time windowed aggregation with watermarks. A stream of events (each with an event-time timestamp) is aggregated into 1-minute tumbling windows (counting events per minute). Events arrive -- possibly out of order (an event that happened at 10:00:30 might arrive after one that happened at 10:00:45) and possibly late. Each event is placed in the window for its event time (the 10:00-10:01 window for the 10:00:30 event -- regardless of when it arrived) -- event-time windowing (correct by occurrence). The watermark tracks the event-time progress (as events arrive, the watermark advances -- asserting 'event time has reached X'). When the watermark passes 10:01 (asserting event time has advanced past the window's end -- so the 10:00-10:01 window's events have probably all arrived), the window is considered complete and its result (the count for 10:00-10:01) is emitted. So the window was aggregated by event time (correct -- events in their occurrence windows), and the watermark determined when it was complete (despite out-of-order arrival) -- correct event-time windowing. The event time plus watermark handled the out-of-order events correctly (windowed by occurrence, emitted when the watermark indicated completeness).
The late-data and window-type vignettes show the handling and choices. A late-data case: an event for the 10:00-10:01 window arrives after the watermark passed 10:01 (a late event -- e.g., delayed by a network issue). With allowed lateness configured (keeping the window state a bit longer past the watermark), the late event is still accommodated (added to the window, and the result updated) -- so the late data isn't lost (within the allowed lateness). Beyond the allowed lateness, a late event would be dropped (or sent to a side output) -- the trade-off (accommodate late data -- more state/latency -- vs drop it -- less state, but incomplete). The allowed lateness handled the late data. A window-type case: the team uses tumbling windows for distinct per-minute counts (each minute a distinct window), sliding windows for a rolling 5-minute average (updated every minute -- overlapping windows), and session windows for user-session grouping (a window per user's activity burst, ending after inactivity) -- choosing the window type per the aggregation semantics needed.
The state and lateness-tuning vignettes complete it. A state case: the windowing maintains state per window (each window's running aggregation) -- which grows with the number of active windows and their data. The team manages the state (cleaning up completed windows -- past the watermark and allowed lateness -- to bound the state; and the allowed lateness affecting how long state is kept -- more allowed lateness, more state) -- bounding the window state. A lateness-tuning case: the team tunes the watermark and allowed lateness -- balancing completeness (a later watermark / more allowed lateness accommodates more late data -- more complete results) against latency (waiting longer to emit -- higher latency) and state (keeping windows longer -- more state) -- choosing the balance for their needs (e.g., more allowed lateness for completeness-critical analytics, less for low-latency needs). The lateness tuning balanced completeness, latency, and state. The consolidated discipline the team documents: use windowing to aggregate unbounded streams (bounding the infinite into finite windows), choose the window type by the aggregation semantics (tumbling for distinct periods, sliding for rolling aggregations, session for activity-based grouping), use event time for correct windowing (by occurrence, not arrival -- since events arrive out of order/late) with watermarks (tracking event-time progress to know when windows are complete), handle late data (allowed lateness -- balancing completeness against latency/state), control emission with triggers, manage the window state (bounding it -- cleaning up completed windows), support windowed joins, and tune window sizing and lateness -- because windowing is the fundamental technique for aggregating unbounded streams (bounding the infinite into finite windows), inseparable from the hard time semantics (event time, watermarks, late data) that make stream processing correct.