Why architecture matters here
Structured Streaming architecture matters because streaming introduces failure modes that batch never had. Late-arriving data can invalidate aggregate results. Restart can miss events or duplicate them. Stateful joins can grow unbounded. Each has a specific architectural remedy — watermarks bound lateness, checkpointing preserves offsets, TTL bounds state — that Structured Streaming provides but you must configure correctly.
Cost matters. Streaming clusters run continuously; every minute of over-provisioning multiplied by hours means real money. Autoscaling, right-sized cluster, and choosing between micro-batch and continuous shape spend materially.
Correctness under lateness is where great streaming architectures differ from mediocre ones. Watermarks + windowed aggregations gracefully handle late events; without them, results silently drift.
The architecture: every piece explained
Walk the diagram top to bottom.
Sources. Kafka, Kinesis, files (auto-detect new files), Rate (test source), Socket (dev only). Each source provides progress markers (Kafka offsets, file listings) for exactly-once support.
Streaming DataFrame. Programming model: an unbounded table where new rows are appended over time. All DataFrame ops (select, filter, groupBy, join) work the same as batch.
Query. The SQL or DataFrame computation to run continuously. Structured Streaming translates it into an incremental execution plan.
Micro-batch vs Continuous. Micro-batch (default) executes small batches every trigger interval — high throughput, second-scale latency. Continuous processing runs one long-lived task per partition — millisecond latency, more limited feature set.
Watermark. Event-time cutoff: "we won't process events older than this." Used for state cleanup and correctness in window aggregations. Setting watermark = max_event_seen - lateness_tolerance.
State Store. Persistent storage for aggregate state, join keys, deduplication sets. Modern default is RocksDB-backed for size and performance. State is checkpointed periodically.
Aggregations + Joins. Windowed aggregations (tumbling, sliding, session windows). Stream-stream joins with watermark on both sides for state cleanup. Stream-static joins (dimension enrichment).
Sinks. Kafka (append), Delta Lake (ACID upserts), JDBC (batch upserts), console (dev), memory (test). Exactly-once requires idempotent or transactional sinks.
Checkpoint + WAL. Checkpoint stores committed source offsets, current state, and metadata. WAL logs each batch commit. On restart, replay from last committed offset.
Exactly-Once. Combination of source offset tracking + idempotent/transactional sink + checkpointing. Works out of the box for Kafka→Delta pipelines.
Adaptive + backpressure + DLT. AQE optimizes streaming queries. Backpressure adjusts source rate based on downstream capacity. Delta Live Tables provides declarative streaming pipelines on top.
End-to-end streaming aggregation
Trace a click-stream aggregation. A pipeline reads from Kafka topic "clicks," computes hourly clicks per campaign, writes to a Delta table.
readStream from Kafka creates a streaming DataFrame with (offset, key, value, timestamp) rows. A withWatermark("event_time", "1 hour") declaration says "tolerate 1 hour of lateness."
groupBy(window("event_time", "1 hour"), "campaign_id").count() defines a windowed aggregate. The query is registered.
writeStream to Delta table with output mode = append. Trigger every 30 seconds. Checkpoint location on S3.
Query starts. Every 30 seconds: read new Kafka messages since last checkpoint; compute deltas to windowed counts; update state store; write finalized windows to Delta; commit new offset to checkpoint.
Late data: an event with event_time from 2 hours ago arrives. Watermark has advanced past that window; the event is dropped (or optionally sent to a late-data output stream).
Query crashes. On restart, it reads the last checkpoint: last committed Kafka offset, current state store snapshot. Resumes from there. Delta transactional write ensures no duplicate rows even if a batch was in-flight during crash.
Result: exactly-once end-to-end aggregation over unbounded stream.