Why architecture matters here
The architecture matters because reading the replication log is fundamentally different from, and vastly better than, the alternatives for keeping an analytics copy fresh. Polling the source with SELECT * WHERE updated_at > last_seen misses deletes entirely, misses rows updated without touching the timestamp, and adds real query load; a full nightly dump is complete but stale and expensive. Log-based CDC captures every change — including deletes — in exact commit order, with negligible source overhead, because the log is already being written for replication. Datastream's whole value proposition rests on consuming that existing artifact instead of interrogating the database.
It matters, second, because it decouples the analytics and transactional workloads cleanly. Once changes flow through Datastream into BigQuery, every analytical query hits BigQuery, not the operational database. The production store sees only Datastream's log reader — a lightweight, sequential consumer — and is otherwise untouched by the analytics tier's scans and joins. That isolation is what lets a company run heavy reporting, machine-learning feature extraction, and ad-hoc exploration on near-live data without ever putting that load on the system of record.
The serverless nature matters because CDC is notoriously fiddly to self-operate. A hand-rolled CDC pipeline (Debezium on Kafka, say) is powerful but demands running connectors, a message bus, sink workers, and the scaling and failure-handling for all of them. Datastream removes that operational surface: Google runs the capacity, handles scaling with throughput, and manages checkpointing and recovery. The tradeoff is less control and a fixed feature set — you get the sources and sinks Datastream supports, configured the way it supports them — but for the common case of 'get my Cloud SQL / self-managed MySQL / Oracle changes into BigQuery,' the managed path removes an enormous amount of undifferentiated toil.
Finally, the backfill-plus-CDC model matters because it is what makes the destination both complete and current. CDC alone only captures changes from the moment it starts, so a table that existed before the stream began would be missing all its historical rows. Datastream first backfills the existing data, then switches to log tailing, and — critically — coordinates the handoff so changes that occur during the backfill are not lost and not double-applied. Getting that boundary right is subtle and is exactly the kind of correctness detail a managed service earns its keep on: the result is a destination that starts as a faithful full copy and stays faithful as changes stream in, which is the entire point of building the pipeline in the first place.
The architecture: every piece explained
The CDC reader is the component that consumes the source's replication log. For MySQL it reads the binary log; for PostgreSQL it uses logical decoding over the WAL (via a replication slot and publication); for Oracle it reads the redo logs. In each case it registers as a log consumer, so the database streams committed changes to it in order. The reader also performs the initial backfill by reading existing table contents, coordinating that snapshot with the log position so the two phases join seamlessly.
The change event is the unit Datastream produces. Each event describes one row-level change — an insert, update, or delete — carrying the row's values and metadata: the source table, the operation type, a commit timestamp, and the log position. Updates carry the new row state; the event stream is an ordered record of every mutation, which is what lets a sink reconstruct the source's current state or retain full history.
Private connectivity is what lets a Google-managed service reach a database that lives inside a private network. Datastream supports IP allowlisting, a forward-SSH proxy, or — the clean option — VPC peering / Private Service Connect, so the connection to the source never traverses the public internet. Because the source is usually a production database behind strict network controls, this connectivity configuration is often the most involved part of setting up a stream.
The stream position is the checkpoint that makes the pipeline resumable and exactly-once at the destination. Datastream persists how far it has read in the source log (the binlog coordinate, the WAL LSN, the redo SCN); if it restarts or a sink is briefly unavailable, it resumes from the last committed position, replaying no more and no less than needed. The BigQuery sink delivers changes into BigQuery tables, merging inserts, updates, and deletes so the destination table mirrors the source in near-real-time (a few minutes of freshness lag), which is the most common target. The Cloud Storage sink instead writes change events as Avro or JSON files to GCS, for pipelines that want to process the raw change stream themselves with Dataflow or load it on their own schedule. Together these pieces — log-reading CDC engine, structured change events, private connectivity, a durable stream position, and the BigQuery or GCS sink — are the whole pipeline, and each addresses a specific requirement of moving live operational data into analytics safely and continuously.
End-to-end flow
Trace a table from empty destination to live mirror. An operator creates a stream: a source connection to a MySQL database over Private Service Connect, a destination of a BigQuery dataset, and a selection of tables including orders. When the stream starts, Datastream first backfills orders — it reads the table's existing rows and writes them into the BigQuery orders table, while noting the binlog position at which the backfill began so it knows where the change stream must pick up.
With the backfill complete, Datastream switches to CDC. It tails the MySQL binlog from the recorded position, and for every committed change to orders it emits a change event: an INSERT of a new order becomes an event with the new row, an UPDATE of an order's status becomes an event with the updated values, a DELETE becomes a delete event carrying the key. Each event is tagged with its commit timestamp and binlog coordinate.
The BigQuery sink applies these events to the destination table. Using the change type and the primary key, it merges each event into orders so that an insert adds a row, an update overwrites the matching row, and a delete removes it — keeping the BigQuery table a faithful, current copy of the source, typically within a few minutes. Datastream advances its stream position as batches are committed, so if the sink or the service restarts, it resumes from exactly where it left off and neither loses nor duplicates changes at the destination.
Now a schema-drift moment, because it shows the pipeline's seams. A developer adds a column promo_code to the source orders table. The next change events carry the new column, and Datastream propagates the schema change to the BigQuery destination, adding the column so subsequent rows populate it. Meanwhile an analyst runs a Looker dashboard and a dbt model that read the BigQuery orders table — they see near-live order data, including the new column, without any of their queries ever touching the production MySQL server. If the source database's binlog retention were too short and Datastream fell behind far enough that the needed log had already been purged, the stream would error and require a re-backfill — which is exactly why log retention and freshness lag are the metrics an operator watches. On the happy path, though, the end-to-end flow is a continuous, self-checkpointing river: the source commits, the log carries the change, Datastream reads and delivers it, and the analytics copy reflects it minutes later, with the transactional store none the busier for it.