Why it matters

Serialization overhead can dominate. Understanding options improves throughput.

Advertisement

The architecture

Configuration: spark.serializer=org.apache.spark.serializer.KryoSerializer.

Register classes: spark.kryo.registrator or SparkConf.registerKryoClasses.

Kryo enablementDefault Javaslow, verboseKryofast, compactRegister classesfor max efficiencyRegistering classes gives ~2x more improvement over unregistered Kryo
Kryo tuning.
Advertisement

How it works end to end

Automatic serialization: works out of box. Registration optional but recommended for max perf.

Custom types: register with Kryo or use built-in support for Scala case classes.

Impact: 2-10x faster serialization, 2-5x smaller output.