Why architecture matters here
The architecture matters because it decides whether your data platform is elastic and cheap or a permanently-on cost center that everyone fights over. The fused compute-storage model of classic Hadoop forces a single long-lived cluster: you cannot turn it off because the data lives on its disks, so you size it for peak, run it 24/7, and watch it sit idle most nights. Every team's job competes for the same YARN queues, a noisy neighbor can starve a critical pipeline, and upgrading the cluster means a coordinated outage for everyone.
Dataproc's separation of storage and compute dissolves those constraints, but only if you actually use it that way. When the durable data lives in GCS, a cluster is just transient compute you rent for the length of a job. You can give each pipeline its own right-sized cluster, run experimental work on preemptible VMs at a fraction of the price, and let clusters autoscale up for a heavy join and back down when it finishes. Cost tracks work done rather than wall-clock time, and isolation is physical — one team's runaway job cannot touch another's cluster because they are different clusters.
Getting this right is an architectural decision about lifecycle. Treat Dataproc as a lift-and-shift of your old always-on cluster — one big cluster everyone submits to — and you inherit all of Hadoop's contention and cost problems with a cloud bill on top. Design instead around ephemeral, job-scoped or workflow-scoped clusters over a shared GCS lake, and you get the elasticity the platform was built to deliver. The choice is made in how you provision, not in the service itself.
There is an organizational payoff beyond the invoice. When compute is disposable and cheap, experimentation stops being a scarce, permission-gated activity. A data scientist can spin up a right-sized cluster to test a new join or a different Spark configuration against real data in GCS, measure it, and tear it down — paying cents, blocking no one, leaving no residue. On a shared always-on cluster that same experiment competes for finite queues and risks destabilizing production pipelines, so it either does not happen or happens fearfully. Elastic, isolated compute shifts the culture around data work from rationing toward iteration, and that shift is often worth more than the direct cost savings.
None of this means Dataproc is always the right tool — it earns its place specifically when you have existing open-source Spark, Hive, or Hadoop workloads, custom code, or ecosystem dependencies you want to run largely unchanged. For pure SQL analytics a warehouse like BigQuery is usually the better fit, and the architectural skill is recognizing that Dataproc's value is portability and control of the open-source stack delivered elastically, rather than being the answer to every data question. Choosing it for a workload that BigQuery would run without any cluster at all reintroduces exactly the operational overhead the platform was meant to shed.
The architecture: every piece explained
The Dataproc API is the control plane: it provisions clusters, submits jobs, and runs workflow templates. A cluster has three node roles. The master runs the YARN ResourceManager, the HDFS NameNode, and the job drivers; a high-availability configuration runs three masters. Primary workers run YARN NodeManagers and HDFS DataNodes and hold both compute and scratch storage. Secondary workers are compute-only and typically preemptible (Spot) VMs — they add task capacity cheaply but can be reclaimed by Google at any time and do not store HDFS data, so losing one costs only in-flight tasks, not durability.
On the cluster run the familiar engines. YARN schedules containers across workers; HDFS provides cluster-local scratch and shuffle space; and Spark, Hive, Flink, or Presto execute the actual work. The critical distinction is that HDFS here is scratch, not the system of record — the durable data lives in Cloud Storage, reached through the Cloud Storage connector so that gs:// paths behave like a filesystem. An autoscaling policy watches YARN's pending-container demand and adds or removes secondary workers within configured bounds, with a graceful-decommission timeout so shrinking does not kill running tasks.
Around the cluster sit the integrations that make it a platform. Connectors let Spark read and write BigQuery tables and Bigtable directly. The Dataproc Metastore is a managed, highly available Hive Metastore shared across clusters, so table definitions survive any single ephemeral cluster's death. Initialization actions are scripts run at cluster creation to install packages or configure the nodes, and the Component Gateway exposes web UIs (Spark History, YARN, Jupyter) securely. Workflow templates bundle 'create cluster, run these jobs in a DAG, delete cluster' into one managed operation.
Two operational details bind a cluster together. Every Dataproc cluster is created from an image version that pins the exact releases of Spark, Hadoop, and the rest of the stack, so a pipeline runs on the same engine versions every time rather than drifting as upstream releases land. And each cluster is given a staging and temp bucket in GCS where the service stores job dependencies, driver output, and Spark history, so that even an ephemeral cluster leaves a durable audit trail and the history server survives the cluster's deletion. Together with initialization actions for custom setup and the Component Gateway for secure UI access, these give you reproducible, auditable, inspectable clusters despite their transience — the cluster is disposable, but the record of what it did is not.
End-to-end flow
Follow an ephemeral batch job. A scheduler or engineer submits a workflow template to the Dataproc API. The control plane provisions a fresh cluster: it boots the master and primary workers, runs any initialization actions, brings up YARN and HDFS, and — if configured — attaches secondary preemptible workers for extra capacity. The cluster registers with the shared Dataproc Metastore so it sees the existing table catalog.
The template's jobs then run in dependency order. Each Spark job's driver launches on the master, requests executors from YARN, and those executors read input data from gs:// paths in Cloud Storage. During execution, intermediate shuffle data is written to local disk and HDFS scratch on the workers — this is why the cluster is not stateless mid-job. As YARN's pending demand rises during a heavy stage, the autoscaler adds secondary workers; as the stage completes and demand falls, it gracefully decommissions them after their tasks drain. Outputs are written back to GCS, and any table metadata changes are recorded in the Metastore.
When the final job in the DAG finishes, the workflow deletes the cluster. The durable results and the catalog survive in GCS and the Metastore; the compute simply disappears, and billing stops. A later job — perhaps a different team's — spins up its own cluster against the same GCS lake and Metastore and sees exactly the data the first job produced. This create-run-delete rhythm, with GCS as the permanent floor beneath transient clusters, is the pattern Dataproc is built to make cheap and repeatable. Long-lived clusters exist too — for interactive notebooks or streaming — but even those read and write GCS so that the cluster's death is never a data-loss event.
It is worth being precise about the state boundary during that run. While the cluster is alive, three kinds of state coexist: durable inputs and outputs in Cloud Storage, transient shuffle and scratch on worker local disk and HDFS, and catalog metadata in the shared Metastore. Only the first and third survive cluster deletion; the second is deliberately expendable, which is exactly why a preempted secondary worker costs only its in-flight tasks and never durable data. This is also why a long job must checkpoint or write partial results to GCS if it wants them to outlive the cluster — anything left only on HDFS scratch vanishes the moment the workflow's delete step runs, by design rather than by accident.