Why architecture matters here
Warehouse DR is architecturally hard for a reason worth stating precisely: consistency spans two storage systems with different clocks. Files land in HDFS/S3; the metadata that makes them queryable commits to an RDBMS; a copy taken of each independently captures a moment that never existed — partitions without files, files without partitions, an ACID table whose delta directories are present but whose transaction watermarks say otherwise. The notification log is what makes principled replication possible: it is a single sequence in which every metadata change (and, via write-ID tracking, every transactional data change) has a position, so 'replicate up to event 84,201' names a consistent warehouse state. REPL DUMP/LOAD is essentially log shipping for a warehouse, with the log being the metastore's change stream.
The event-driven design matters operationally because the alternative — periodic full comparison — cannot scale. Diffing a million-partition warehouse against its replica is itself a heavy distributed job, and it discovers what changed without knowing in what order — and order is everything when a table was dropped and recreated with a different schema between cycles. Incremental replication's cost scales with change rate, not warehouse size; a quiet weekend cycle ships kilobytes. The trade is a dependency chain you must protect: if the notification log's retention is shorter than your longest replication outage, the incremental chain breaks and the answer is a new bootstrap — of everything. Log retention versus worst-case lag is the first arithmetic of any deployment.
Finally, replication policy is business architecture, not just plumbing. Compliance may require some databases replicated cross-region and others forbidden from leaving one; the DR site's capacity may afford the curated warehouse but not the raw landing zone; RPO targets may differ by tier (finance tables: 15 minutes; sandbox: none). The per-database policies, table filters, and cadence schedules are where those decisions live — and where auditors will look.
The architecture: every piece explained
Top row: the change pipeline. The primary warehouse generates events — CREATE/ALTER/DROP, add-partition, insert, ACID commit — into the notification log, each with a monotonically increasing ID. REPL DUMP dbname in bootstrap mode serializes the whole database (metadata objects plus, for managed tables, data-file references); in incremental mode it reads events from the last-replicated ID to current, filtering by the replication policy (database scope, table include/exclude regexes), and writes an ordered changeset to the dump directory: metadata operations as serialized events, data changes as file manifests — the dump is instructions, not (necessarily) the data itself.
Middle row: moving and applying. Data copy executes the manifests: DistCp for HDFS-to-HDFS (with snapshot-based consistency where configured), or object-store transfer for cloud targets; copies can run lazily (load pulls files) or be prestaged. REPL LOAD dbname FROM path applies the changeset on the target in event order — creating tables, adding partitions pointing at the copied files, applying drops and renames — idempotently, so a failed cycle re-runs from its checkpoint rather than corrupting. The DR warehouse stays read-only for consumers (reporting can run against it — a common secondary payoff), with its own metastore tracking replication status per database (last event ID applied — the replica's version vector).
Bottom rows: the hard parts and the harness. ACID handling: transactional tables replicate with their write-ID semantics — the dump captures committed transactions up to a boundary, open transactions at dump time are excluded (they'll ship next cycle after commit), and the target's tables get valid write-ID state so snapshot reads behave; compaction on the primary is an event like any other. External tables replicate metadata by default with data movement as a separate policy decision (often the data pipeline re-lands them at DR independently). Orchestration is yours to own: a scheduler runs dump→copy→load per database on its cadence, persists checkpoints, retries idempotently, and alerts on cycle failure. The ops strip closes the loop: replication lag (events and minutes) per database against RPO, dump/copy/load durations against the cycle budget, and — the part everyone skips — rehearsed failover and failback runbooks.
End-to-end flow
Stand up DR for a 400-table finance warehouse, then survive a failover. Day one: bootstrap. REPL DUMP finance serializes the database — metadata for every table, manifests for 200TB of managed ORC. The copy tier moves files over three days (bandwidth-throttled to protect production); REPL LOAD finance on the DR cluster applies — databases, tables, partitions, ACID write-ID state — and records the bootstrap's ceiling event ID: 1,204,556. From here, a scheduler runs the incremental cycle every 30 minutes: dump events since checkpoint (typically a few thousand — partition adds from ETL, a handful of ALTERs, ACID commits), copy the new files (tens of GB), load in order, advance the checkpoint. Steady-state lag: 35–50 minutes — inside the 1-hour RPO the business signed.
The cycle's texture shows in its edge handling. A dump catches an ETL MERGE mid-flight: the open transaction is excluded; its commit event ships next cycle — the replica only ever shows committed states. A table is dropped and recreated with a new schema between cycles: the events apply in order — drop, create, add partitions — so the replica never holds a hybrid. An analyst's sandbox table matching the exclude pattern churns violently: zero replication cost, by policy. A network blip fails a copy at 60%: the cycle's load never ran, the checkpoint never advanced; the retry re-executes idempotently.
Failover day: the primary region's storage layer fails hard at 09:14. The runbook executes: confirm the replica's last applied checkpoint (09:02 — RPO loss: 12 minutes of events, enumerable from the primary's log when it returns); flip the DR warehouse to read-write; repoint BI and pipeline configs via the prepared connection aliases; declare the ETL restart point from the checkpoint's event position. Reporting resumes within the hour. Failback, two days later, is replication reversed: the (former) DR is now primary and has diverged; a reverse bootstrap of changed databases (or, for the disciplined tables, reverse incremental from the failover point) rebuilds the original site; a scheduled cutback window swaps roles again. The whole sequence worked because it had been rehearsed quarterly against a scratch database — the rehearsal being, as always in DR, the actual product.