Core concept

spark.readStream.format("kafka")
  .option("kafka.bootstrap.servers", "broker:9092")
  .option("subscribe", "topic")
  .load()
Advertisement

How it works

Offset management via checkpoint. startingOffsets for initial position.

Advertisement

Trade-offs + gotchas

Every Kafka-Spark pipeline.