Why architecture matters here
Dataflow fails on cost surprises, wrong worker sizing, and streaming engine misuse. Architecture matters because autoscale + shuffle + windowing decide behavior.
The architecture: every piece explained
The top strip is the runtime. Beam pipeline SDK. Dataflow runner translate + execute. Streaming engine — offload state. Autoscaler workers.
The middle row is streaming. Shuffle service managed. Watermark + timers event time. Prime containers — Flex Templates. Snapshot / update in-place.
The lower rows are ops. IO connectors — Pub/Sub + BQ + GCS. Metrics + jobs UI. Ops — cost + drain + templates.
End-to-end flow
End-to-end: Beam pipeline reads Pub/Sub, windows events, writes to BQ. Dataflow runner picks streaming engine. Autoscale adjusts to 20 workers. Watermark + timers handle late data. In-place update deploys new version.
Dataflow as a managed Beam runner
A Dataflow job is not written against Dataflow. It is written against the Apache Beam model - a directed graph of PTransforms connected by PCollections, where a PCollection is an unordered, potentially unbounded bag of timestamped elements and a PTransform is a named node that consumes one and produces another. Beam deliberately says nothing about machines. It specifies what the graph computes, when results are emitted relative to event time, and how state is keyed - and it leaves execution entirely to a runner. The general Beam programming model, its transform vocabulary and its portability layer are covered separately in the streaming section; what follows is only the part of the story the model refuses to specify, which is exactly the part Dataflow supplies.
When you call pipeline.run() with the Dataflow runner, the SDK does not start executing anything. It serialises the graph, uploads your dependencies to a staging bucket in Cloud Storage, and submits a job description to the Dataflow service. The service is the thing that optimises the graph, provisions a fleet of Compute Engine worker VMs in your project, distributes work to SDK harness processes running inside containers on those workers, monitors progress, and tears the fleet down. Your code runs on VMs you are billed for but never ssh into by default; the control plane that decides how many of them exist is Google's.
The portability claim is real but narrower than it sounds. The same Beam source can run on the Flink runner, the Spark runner or the direct runner, and the transform semantics will hold. What does not port is everything around the transforms: the IO connectors that matter most on Google Cloud (Pub/Sub, BigQuery, Bigtable, Spanner) have their most complete implementations on Dataflow, the autoscaling behaviour is entirely runner-specific, and the operational surface - update, drain, snapshots, per-stage metrics - is a Dataflow feature set with no equivalent on another runner. Treat portability as insurance on your business logic, not as a promise that the pipeline will behave identically elsewhere.
Fusion - why your DoFn boundaries are not execution boundaries
This is the single most important thing to understand about Dataflow, and it is invisible in your source code. Before execution the service runs a graph optimiser, and its main move is fusion: adjacent element-wise transforms that do not require a data redistribution are collapsed into a single execution stage. A chain of four ParDos becomes one stage. Each element is pulled through all four function bodies in the same thread on the same worker, with no serialisation, no materialisation of the intermediate PCollections, and no network hop between them. That is why Beam pipelines can afford to be written as many small, readable, single-purpose transforms - the cost of the abstraction is optimised away.
The consequences bite in three places. First, parallelism is a property of the stage, not of your transform. If a fused stage begins with a source that produces four splits - four files, four Pub/Sub shards, one unsplittable compressed object - then every transform fused into that stage runs at that parallelism, no matter how expensive it is. Adding workers does nothing, because there is no work to hand them. Second, per-step counters and wall-clock attribution are approximate inside a fused stage, because there is no boundary at which the runner can measure. The job UI will show you stage-level timing that does not decompose cleanly onto the steps you wrote. Third, a fused stage fails and retries as a unit: an exception in the fourth transform re-runs the element through the first three, so any side effect in an earlier DoFn must be idempotent.
The remedy is a deliberate fusion break. Anything that forces a redistribution - a GroupByKey, a Combine with a shuffle, or the explicit Reshuffle transform - ends a stage and lets the next one be scheduled independently, at a parallelism the runner chooses. The classic shape is a source that yields a small number of elements which each fan out into a large amount of work:
import apache_beam as beam
from apache_beam.transforms.util import Reshuffle
with beam.Pipeline(options=opts) as p:
(p
# A handful of manifest files -> only a handful of splits.
| "ReadManifests" >> beam.io.ReadFromText("gs://bucket/manifests/*.txt")
# Each manifest line names thousands of records to fetch.
| "ExplodeRefs" >> beam.FlatMap(expand_manifest)
# WITHOUT this, ExpensiveCall is fused to ReadManifests and runs
# at manifest-file parallelism. Reshuffle ends the stage.
| "FusionBreak" >> Reshuffle()
| "ExpensiveCall" >> beam.ParDo(EnrichFromApi())
| "WriteBQ" >> beam.io.WriteToBigQuery(table, method="STORAGE_WRITE_API"))
A fusion break is not free - it is a real shuffle, with real serialisation and real billed bytes. Insert one where the fan-out ratio justifies it, not everywhere. The diagnostic signal is a job that pins a small number of workers at high CPU while the autoscaler refuses to add more: that is almost always a fused stage inheriting source parallelism, not a shortage of capacity.
Dataflow Shuffle and Streaming Engine - moving state off the worker
By default a distributed data processor keeps two heavy things on the worker that runs the code: the shuffle data produced between stages, and, for streaming, the keyed state and timers that windows and stateful DoFns accumulate. Dataflow's two service-side offloads exist to take both away. The decoupling of shuffle from executors as a general architectural idea is developed at length in the Spark and Hadoop shuffle-service articles; the point here is what changes specifically for a Dataflow job.
Dataflow Shuffle is the batch offload. The shuffle for a batch job is performed in the service backend rather than on worker persistent disks. The worker VMs stop being stateful with respect to intermediate data, which has three downstream effects: worker disks can be small because they no longer hold spilled shuffle output, the autoscaler can remove a worker mid-job without triggering a redistribution of shuffle data that would have to be re-fetched or recomputed, and a worker that dies takes only its in-flight work with it rather than a partition of the shuffle.
Streaming Engine is the streaming equivalent and goes further: window state, keyed state and timers live in the service, not in the worker's memory and disk. That inverts the sizing problem. Without it, a streaming job's worker count is bounded from below by how much state you hold, because the state has to fit somewhere - so a job with large session windows needs big workers even when its CPU demand is trivial, and shrinking the fleet means moving state between workers. With Streaming Engine, worker count tracks CPU and backlog alone, and adding or removing a worker is a reassignment of key ranges rather than a migration of gigabytes.
Both are billed by data processed, which is a separate line from the worker VMs. That is the tradeoff to actually reason about: you are trading persistent disk and worker-hours for a per-byte service charge, and for a shuffle-heavy job the per-byte charge can be the larger number. It usually still wins, because the alternative is provisioning worker disk for peak shuffle volume and paying for it for the whole job.
Horizontal autoscaling and dynamic work rebalancing
Dataflow's streaming autoscaler watches two signals. The first is backlog expressed in time - roughly, at the current throughput, how long would it take to drain what is queued at the source - together with whether that backlog is growing or shrinking. The second is CPU utilisation across the current fleet. Backlog alone is not enough, because a job can be keeping up with a small steady backlog while running its workers flat out, and a job can have zero backlog because its source is quiet rather than because it is fast. The autoscaler adjusts worker count within the bounds you set, and the upper bound is a real ceiling, not a hint: a job pinned at its maximum with a growing backlog will fall behind indefinitely and will not tell you it wanted more.
The reason this autoscaler can be aggressive is the previous section. Because Streaming Engine holds the state, scaling down is cheap - key ranges are reassigned rather than state being drained off a departing worker. Runners that keep state worker-local have to make downscaling a much more careful, much slower operation, and in practice tend to be tuned to avoid it.
Batch gets a different mechanism: dynamic work rebalancing. A batch job's input is split into bundles up front, but splits are rarely uniform - one file is ten times the size of the others, one key is skewed, one worker's VM is simply slower. Rather than let the job's completion time be set by its slowest split, the service can re-split remaining work from a straggling worker and hand a portion of it to an idle one, mid-execution. This is why a Dataflow batch job's tail behaves better than a naive partitioned job, and it is also why the parallelism you observe does not match the file count you started with.
Three things defeat the autoscaler regularly, and none of them are fixed by raising the maximum. A non-splittable source - a single gzip object, a JDBC read without a partition column - caps parallelism at the source. A fused stage, per the previous section, hides available parallelism behind an upstream bottleneck. And a throttling sink - a downstream API, a hot Bigtable row range, a quota-limited endpoint - produces exactly the backlog signature that makes the autoscaler add workers, which increases pressure on the sink and makes things worse. Read the sink's own latency before you read the worker count.
Event time in a Dataflow job - the operational surface
Event-time semantics are not a Dataflow invention and are not covered here. What a watermark is, how it evicts window state, how late rows are dropped and how to choose the allowed lateness are developed in depth in spark_streaming_watermark, and the fixed / sliding / session window taxonomy belongs to streaming_windowing. Read those for the concepts. What is worth saying about Dataflow specifically is where those concepts surface operationally.
The watermark in Dataflow is per stage, not per job. Each fused stage has its own input and output watermark, and a stage's output watermark cannot advance past its input watermark minus whatever the stage is still holding. This is what makes a stuck job diagnosable: you look down the stage list for the first place the watermark stops advancing, and that stage - not the job as a whole - is where the buffered work or the stuck key lives.
For a Pub/Sub source, the watermark cannot be read off the data, because Pub/Sub does not deliver in event-time order. The service estimates it, and the estimate is bounded by the oldest message still unacknowledged on the subscription: until that message is processed, the watermark cannot safely move past its timestamp. The practical consequence is that a single poison message that keeps failing and being redelivered will freeze the watermark for the whole pipeline, and every downstream window will stop emitting. The symptom is not an error rate, it is silence. A dead-letter path is not a nicety here; it is what keeps the watermark moving.
Two metrics are the ones to alert on. Data freshness is the gap between now and the watermark - how far behind event time the pipeline is. System lag is how long the oldest in-flight element has been being processed. They fail differently: rising system lag with flat freshness usually means a slow external call inside a DoFn, while rising freshness with normal system lag usually means the watermark is being held by something that is not moving at all.
Update, drain and cancel - redeploying a streaming pipeline
A streaming pipeline that has been running for a week holds state you cannot recreate: open session windows, deduplication sets, accumulating combines, pending timers. Redeploying it is therefore a state-migration problem, and Dataflow gives you three distinct answers with three distinct costs. Choosing the wrong one silently loses aggregates.
Cancel stops the job immediately. In-flight work is abandoned, buffered state is discarded, and unacknowledged Pub/Sub messages return to the subscription. It is correct only when the job is broken and its output is already suspect.
Drain stops the job reading new input and then advances the watermark to effectively infinity, which causes every open window to be treated as complete and fire, every timer to run, and every buffered aggregate to be written to the sink. The job then finishes cleanly. Nothing computed is lost - but the semantics change at the boundary: windows that were only partially filled emit as if they were complete, so the final results at the seam are legitimately partial aggregates, not wrong ones. Drain also leaves a gap: the new job starts from wherever the subscription is when it comes up, and the interval between drain completing and the replacement being ready is unprocessed backlog.
Update replaces the running job's graph in place, carrying the state across. The service must be able to prove the new graph can consume the old state, and the identity it uses is the transform name - the string you passed as the step label. Rename a stateful step and the service cannot map its state and will reject the update; change a step's coder or the type of a PCollection incompatibly and it will reject it too. If you deliberately restructured the pipeline, a transform name mapping lets you tell the service that the old name and the new name are the same logical step. The discipline this implies is worth internalising early: step labels in a streaming Beam pipeline are part of your deployment contract, not documentation. Treat renaming one as a breaking change.
Update is what you want for a routine code change; drain is what you want when the graph changed too much for update to accept, and you can tolerate a seam. Snapshots of streaming job state exist as a separate mechanism for restarting from a captured point rather than from a live job.
The cost model, honestly
Dataflow does not have one price, it has three that move independently, and pipelines get expensive in ways the worker count does not reveal.
The first is the worker fleet: vCPU, memory and persistent disk, billed per second for however many VMs the autoscaler decided to run. This is the number people watch, and it is the one the offload features are designed to reduce. The second is data processed by Shuffle and Streaming Engine, billed per byte and completely decoupled from the fleet. A pipeline that is cheap in workers can be expensive here if it groups by key repeatedly, carries fat elements through shuffles, or reshuffles more than it needs to. The third is everything the pipeline touches: Pub/Sub delivery, BigQuery storage-write throughput, Cloud Storage operations, and any external API you call per element. For a high-volume enrichment pipeline the third category routinely exceeds the first.
Four specific traps recur. A streaming job is always on - it holds its minimum workers through the quietest hour of the night, so a low-volume stream can cost more per useful record than an hourly batch job doing the same work. Reshuffle is billed, so a fusion break inserted defensively at every step converts a CPU problem into a data-processed problem. Oversized elements multiply through every shuffle they cross, so trimming fields before a GroupByKey is a direct cost reduction, not a micro-optimisation. And Spot VMs help batch and not streaming: a batch job can absorb preemption through retries, while a streaming job's whole value proposition is continuity.
The honest comparison is against the alternative that has no fleet at all. If the transformation is expressible in SQL over data already landed in BigQuery, a scheduled query has no worker cost, no autoscaler and no state to migrate on deploy. Dataflow earns its price when the work is genuinely streaming, genuinely stateful, or requires code that does not fit in SQL.
When Dataflow, and when something else
Versus Dataproc. These are not competitors so much as two answers to different questions. Dataproc gives you a cluster running the open-source stack - Spark, Hive, Flink - with the versions pinned and the cluster's lifecycle in your hands; you choose it when you have existing Spark or Hadoop code, ecosystem dependencies, or a need to control the engine. Dataflow gives you no cluster at all, only a job, and in exchange you write to the Beam model. If your team already has a Spark codebase, porting it to Beam to gain a managed autoscaler is rarely worth it. If you are starting from nothing and the workload is streaming, Dataflow removes the entire cluster-operations problem that Dataproc merely makes ephemeral.
Versus a direct Pub/Sub subscription. If all you need is to move messages from a topic into BigQuery, possibly with a schema mapping, a BigQuery subscription does that with no pipeline to deploy, no worker fleet, and no state to migrate. Reach for Dataflow when you need something the subscription cannot express: joining two streams, aggregating over event-time windows, deduplicating across a horizon, enriching from a side input, or writing to a sink that is not a simple table.
Versus Cloud Run or Cloud Functions. A per-message stateless transform triggered by a push subscription is cheaper, simpler and faster to deploy than any pipeline. The line is state. The moment correctness depends on seeing several messages together - a window, a session, a running total, a dedup set - you need a system that owns keyed state and event time, and hand-rolling that on top of a request handler and an external store reproduces the hard parts of Dataflow badly.
Versus a self-managed Flink cluster. Flink gives you finer control over checkpointing, state backends and scheduling, and you keep it if you already have the operational muscle. Dataflow's argument is that it removes the cluster and the state store from your responsibility surface entirely, at the price of a per-byte bill and less control over the runtime.