Why architecture matters here
Beam fails on model misunderstandings — thinking batch and streaming differ more than they do. The architecture matters because unified model is the win.
The architecture: every piece explained
The top strip is the model. Pipeline (SDK) in Java/Python/Go. PCollection is the immutable dataset (bounded or unbounded). Transforms like ParDo + GroupByKey. Runner executes on target runtime.
The middle row is event time. Windowing — fixed, sliding, session — groups events by time. Triggers decide when to emit. Watermark tracks completeness. State + timers for user-managed logic.
The lower rows are extras. Side inputs broadcast small data. Observability via metrics + jobs UI. Ops covers schema evolution + runner tuning.
End-to-end flow
End-to-end: pipeline aggregates events per session window. Trigger fires on session close. Watermark ensures completeness. Runner (Dataflow) autoscales workers. State stores per-user counters. Side input holds config. Pipeline works on batch fixtures + live streams identically.