Core concept

val stream = spark.readStream.format("kafka")
  .load()
  .selectExpr("CAST(value AS STRING)")
stream.writeStream.format("parquet").start()
Advertisement

How it works

Micro-batch latency 100ms+. Continuous mode limited support.

Advertisement

Trade-offs + gotchas

Modern Spark streaming. Kafka + Delta + Iceberg sinks.