Core concept
val lookup = spark.sparkContext.broadcast(Map(1 -> "one", 2 -> "two"))
df.map(row => lookup.value(row.getInt(0)))Advertisement
How it works
Broadcast size limits (default 8MB per task). Larger tables use broadcast join hint.
Advertisement
Trade-offs + gotchas
Lookup tables, config, small models.