Why it matters

Wrong join strategy dominates slow queries. Understanding broadcast enables better plans.

Advertisement

The architecture

Small side (below threshold): broadcast to all executors.

Large side: not shuffled. Each executor joins its partition with the broadcast map.

Broadcast joinSmall tablebroadcast to allLarge tableno shuffle neededLocal joinon each executorspark.sql.autoBroadcastJoinThreshold default 10MB; can be increased
Broadcast semantics.
Advertisement

How it works end to end

Configuration: spark.sql.autoBroadcastJoinThreshold. Default 10MB; increase for larger workloads.

Hint: broadcast() hint forces broadcast even if planner didn't choose.

Cost: broadcasting large tables OOMs executors.