Spark Streaming — Micro-Batch Streaming on Spark

Why it matters

Spark Streaming is often the right choice when Spark ecosystem integration matters. Understanding the trade-offs guides tool selection.

The architecture

Micro-batch: collect events for N ms, then process as batch. Standard batches inside Spark.

Structured Streaming: DataFrame API. Continuous mode experimental for lower latency.

How it works end to end

Trigger interval: sets micro-batch size. Trade latency for throughput.

Watermarks: for event-time and late data. Similar to Flink.

State: maintained across batches. Snapshotted to checkpoint.

Sources / sinks: Kafka, files, sockets, Kinesis. Foreach for custom.