Core concept
Driver memory small (2-8GB usual). Executor memory large. Driver can OOM on collect() of big data.
Advertisement
How it works
Driver failure = job dies. Executor failure = task reschedule.
Advertisement
Trade-offs + gotchas
Every Spark app.