Why it matters
Every Spark operational issue traces to one of these components. Driver crashes kill the job; executor deaths lose partitions; cluster manager misconfiguration wastes resources. Knowing the mapping is what makes Spark ops fast.
The architecture
The driver runs your Spark application main method plus the SparkContext or SparkSession that connects to the cluster. It holds the DAG scheduler (turns jobs into stages), the task scheduler (turns stages into tasks and dispatches them), and the block manager master (tracks cached data locations).
Executors are JVMs launched by the cluster manager on worker nodes. Each has a fixed number of cores and memory. Tasks run on cores as threads within the executor JVM. Each executor also has a block manager for cached RDD partitions and shuffle files.
How it works end to end
Application submission: spark-submit launches the driver either on the client host (client mode) or inside the cluster (cluster mode). The driver contacts the cluster manager to request executor containers with specified memory and cores.
Executor lifecycle: cluster manager launches executor JVMs on worker nodes. Each executor registers with the driver. Tasks flow from driver to executors, results flow back.
Dynamic allocation: executors can be added or removed based on workload. Useful for shared clusters and long-running Spark sessions.