HBase Write-Ahead Log — Durability Foundation and Sync Semantics

What the log is, physically

A WAL is an ordinary HDFS file, written by one RegionServer, holding the edits of every region that server hosts. It lives under a per-server directory named for the server's identity — host, RPC port, and start code — so two incarnations of the same host never share a log directory:

/hbase/WALs/rs7.example.com,16020,1717430400123/
    rs7.example.com%2C16020%2C1717430400123.1717430400500       # active
    rs7.example.com%2C16020%2C1717430400123.1717430988000       # rolled, still needed
    rs7.example.com%2C16020%2C1717430400123.meta.1717430400600  # hbase:meta's own WAL

/hbase/oldWALs/                 # archived: no longer needed for recovery
/hbase/data/ns/tbl/<region>/recovered.edits/   # split output, replay input

Three properties fall out of that layout and explain most of the WAL's behaviour. It is shared per server, not per region, so a write to any of a thousand regions is one sequential append rather than one of a thousand open output streams — the trade that log splitting pays back at recovery time. It is on HDFS, not on local disk, so the log is already replicated across DataNodes before anyone thinks about fsync, and a RegionServer's death does not take its log with it. And it is append-only with no in-place update, which is why a WAL is retired by deletion rather than truncation, and why retention is a bookkeeping problem rather than a space-reclamation one.

The hbase:meta region is deliberately carved out onto its own log so that recovering the catalog never queues behind a user table's edits. That separation is configurable in its own right, and it is the first hint that "the WAL" is really a pluggable subsystem rather than one file.

WALKey and WALEdit: what one entry actually holds

An entry is a pair. The WALKey is the envelope: the encoded region name, the table name, a monotonically increasing sequence id, the write time, and a list of cluster ids. The WALEdit is the payload: the cells of a single row mutation, kept together so that the atomicity HBase promises for one row survives a crash — an entry is never half-replayed, because it is never half-written.

The sequence id is the part that does the most work. Each region records the highest sequence id it has flushed into an HFile, so any entry at or below that watermark is already durable and can be skipped during replay. That one comparison is what makes replay idempotent, what makes a re-run split safe, and what decides whether a rolled log file is still needed. The cluster id list is smaller but equally load-bearing: it records which clusters have already seen an edit, which is how a cyclic replication topology avoids shipping the same mutation forever.

Not every entry is a user mutation. HBase writes meta edits into the same stream — flush markers recording that a MemStore was persisted, region-event markers for open and close, compaction descriptors, and bulk-load descriptors naming the HFiles a bulk load adopted. They carry a reserved column family so ordinary consumers can recognise and skip them, and they exist because anything that changes a region's durable state has to be ordered against the mutations around it. Region replicas, in particular, depend on those markers to know when to drop the memory they were tailing.

Providers: filesystem, asyncfs, and multiwal

hbase.wal.provider selects the implementation, and hbase.wal.meta_provider does the same for the catalog log independently. The choice is not cosmetic — it changes the concurrency model of every write on the server.

<property>
  <name>hbase.wal.provider</name>
  <value>asyncfs</value>      <!-- asyncfs | filesystem | multiwal -->
</property>

<!-- multiwal only: how many logs per server, and how regions map onto them -->
<property><name>hbase.wal.regiongrouping.numgroups</name><value>4</value></property>
<property><name>hbase.wal.regiongrouping.strategy</name><value>bounded</value></property>

filesystem is the classic FSHLog: it writes through the standard HDFS output stream, which serialises writers behind the stream's own lock and hands sync work to a small pool of syncer threads. asyncfs is the modern default in HBase 2.x: it speaks to the HDFS write pipeline directly over a non-blocking network layer, so appends do not contend on that lock and a slow replica stalls one pipeline rather than the whole writer. On the same hardware it typically sustains higher append throughput at a lower sync p99, which is why it became the default rather than an option.

multiwal is orthogonal: it keeps a handful of logs per server and maps regions onto them by a grouping strategy — a bounded number of groups, one per region, or one per namespace. More logs means more parallel sync streams and less head-of-line blocking between regions, at the cost of more files to roll, retain, split, and tail. It is a lever for servers hosting a large region count on fast storage, not a default to reach for.

Sync semantics: group commit, not one fsync per put

The single most common misreading of the write path is that each acknowledged mutation buys its own trip to storage. It does not. Handlers append their entries into a shared buffer and then wait on a sync point; a dedicated syncer picks up everything published so far and issues one durability call covering all of it. Fifty concurrent writers on a busy RegionServer are satisfied by one pipeline operation, so the per-write cost of durability falls as concurrency rises — which is exactly why WAL sync latency looks flat under load right up until the storage layer is genuinely saturated.

Two knobs shape that. hbase.regionserver.hlog.syncer.count sizes the syncer pool on the filesystem provider: too few and syncs queue behind each other, too many and you multiply pipeline round-trips that would have batched. hbase.regionserver.optionallogflushinterval sets the cadence for deferred syncing, which is what an ASYNC_WAL table is actually buying — a background sweep on an interval rather than a sync on the critical path, and therefore a loss window bounded by that interval.

What the sync call means physically — bytes in the memory of several DataNodes versus bytes forced onto their disks — is the subject of WAL durability levels, and is chosen per table or per mutation rather than per cluster. The mechanism described here is identical at every level; only the strength of the call at the end of it changes. That separation is worth holding onto: batching is a throughput property of the log, durability is a policy property of the write.

Sequence ids, MVCC, and when an edit becomes visible

Durability and visibility are separate events, and the WAL sits between them. A batch of mutations takes the row locks it needs, starts an MVCC transaction that stamps it with a write number, applies the cells into the MemStore, appends the corresponding entries to the WAL, waits for the sync, and only then advances the MVCC read point past that write number. Scanners filter on the read point, so the cells are physically in memory for a short window during which no reader can see them.

That ordering is what makes the guarantee airtight in both directions. A reader can never observe an edit that is not yet durable, because visibility trails the sync. A reader can never observe half a batch, because the read point advances once for the whole transaction. And if the append or the sync fails, the batch is rolled back out of the MemStore before anything is published — the log is the arbiter, and memory is corrected to match it, never the other way round.

The sequence id in the WALKey and the MVCC write number are the same monotonic counter viewed from two sides: one orders entries in the log for replay, the other orders visibility for readers. Older descriptions of HBase place the MemStore insert strictly after the sync; the observable guarantee is the same, but the modern ordering matters when you are reading a profile, because it explains why MemStore insert time and WAL sync time overlap instead of stacking. For how that memory is then flushed and how backpressure protects it, see the write path.

Rolling: when a log gets cut

A log is closed and a fresh one opened — a roll — for four distinct reasons, and telling them apart is most of WAL triage.

Size. hbase.regionserver.hlog.blocksize sets the block size used for WAL files, deriving from the filesystem default when unset, and hbase.regionserver.logroll.multiplier expresses the roll threshold as a fraction of it. Keeping a log inside a block is deliberate: it bounds how much of a file a split worker has to stream and keeps the write pipeline on one set of DataNodes.

Time. hbase.regionserver.logroll.period forces a roll on a schedule even on an idle server, so a low-traffic region's edits do not sit in a never-closed file indefinitely — which matters to replication, which tails closed files far more comfortably than open ones.

Degraded replication. If the write pipeline drops below hbase.regionserver.hlog.tolerable.lowreplication, the server rolls to obtain a fresh, fully-replicated pipeline rather than continuing to write a log it cannot trust. hbase.regionserver.hlog.lowreplication.rolllimit caps how many times it will chase that in a row, because a cluster-wide DataNode shortage would otherwise turn into a roll storm.

By hand. roll_wal_writer 'rs7.example.com,16020,1717430400123' in the shell cuts the active log on demand — the standard first move when you need replication to finish shipping a file, or when you want a stuck writer replaced without bouncing the server. A spike in rollRequest with a matching spike in lowReplicaRollRequest is a storage-layer incident wearing a WAL costume.

Retention: why you have 900 WAL files

A rolled log is not garbage the moment it closes. It stays live until every region with edits in it has flushed past those sequence ids. One cold column family taking a trickle of writes can therefore pin dozens of otherwise-finished files, because its unflushed watermark sits somewhere back in the oldest of them. This is the single most common WAL pathology, and it is not a bug — it is the retention rule working exactly as written.

HBase defends itself twice. hbase.regionserver.maxlogs bounds how many live logs a server tolerates; crossing it triggers forced flushes of precisely the regions holding the oldest sequence ids, which is why an operator sometimes sees flushes on a region nobody is writing to. hbase.regionserver.optionalcacheflushinterval attacks the same problem from the other end, periodically flushing MemStores that have simply grown old rather than large.

Once a file is genuinely unneeded it moves to /hbase/oldWALs/ — archived, not deleted, because other subsystems may still be reading it. The master's LogCleaner chain decides when it truly goes: hbase.master.logcleaner.plugins lists the vetoes, and a file survives while any of them objects. The time-based cleaner holds files for hbase.master.logcleaner.ttl; the replication cleaner holds any file a peer has not finished shipping. A disabled or unreachable replication peer therefore grows oldWALs without bound, and the resulting HDFS alarm is usually the first symptom anyone notices.

The consumers that are not recovery

Recovery is the WAL's reason for existing, but it is no longer its only reader, and the other consumers are what make WAL health a cluster-wide concern rather than a per-server one.

Replication tails the log. Each RegionServer runs a source per peer that reads entries — including from the log currently being appended to, which is how sub-second cross-cluster lag is possible — filters them by scope, and ships batches to sinks. This is also the coupling that turns a lagging peer into a retention problem: see HBase replication for the shipping path, filters, and serial ordering.

Region replicas can be fed the same stream. With asynchronous WAL replication enabled, a primary's entries are pushed to its secondaries so they can serve timeline-consistent reads from memory rather than waiting for the next flush; the flush and region-event markers described earlier are how a secondary knows to release what it has been holding. See region replicas.

Incremental backup reads archived logs to build the deltas between full images — the reason a backup strategy has an opinion about oldWALs retention. See backup and restore.

Coprocessors can observe the log directly: a WALObserver sees entries before and after they are written, which is the supported hook for auditing or for rewriting edits in flight. See coprocessors for the execution model and its failure modes.

Making entries smaller: compression and encryption

WAL entries are verbose by construction. Every cell repeats its row key, column family, and qualifier, and the entry repeats the table and region identity — redundancy that costs pipeline bandwidth on every write and read time on every replay.

hbase.regionserver.wal.enablecompression turns on dictionary compression: the writer maintains per-log dictionaries for the repeated components and emits short references instead of the full bytes. On a wide-row schema with long qualifiers this is a large reduction; on a schema with short keys and large opaque values it is close to free, because the values were never the redundant part. Later releases added value compression as a separate switch with a selectable codec, aimed at exactly that second case. Both trade CPU on the write path — and on the replay path, where a split worker now decompresses everything it reads — for smaller files and less pipeline traffic.

Encryption is the other transform. With hbase.regionserver.wal.encryption enabled and a key provider configured via hbase.crypto.keyprovider, entries are encrypted at rest in the log, closing the gap left by encrypting only HFiles: without it, recent mutations sit in plaintext on HDFS until they are flushed. The operational catch is key availability at the worst possible moment — a recovering RegionServer cannot replay a log whose key it cannot fetch, so the key provider becomes part of the recovery path and needs the availability of one.

What happens when the server dies

The full recovery path belongs to WAL splitting, but the shape of it is worth carrying here because it explains why the choices above matter.

The master notices the RegionServer's ZooKeeper session expire, marks it dead, and moves its WAL directory somewhere no one will append to it. Because the log interleaves every region's edits, it must be demultiplexed: split workers read the files, group entries by region, skip anything at or below each region's last-flushed sequence id, and write per-region recovered.edits files. Regions are then reassigned, and each one replays its file into a fresh MemStore before it serves a single request.

Everything in this article shows up in that window. More unflushed WAL means more to split, so the retention pathology above is also a mean-time-to-recovery problem. Compression means the split worker spends CPU decompressing. A multiwal server has more files to split but more parallelism available to do it. And a log that was rolled promptly is a log that is already closed and cheap to read.

One piece of history is worth knowing because it still appears in old runbooks: HBase once offered distributed log replay, which sent entries straight to the recovering RegionServers instead of materialising recovered.edits files. It was removed in 2.0, and modern recovery is distributed log splitting — coordinated through the procedure framework — plus replay. Configuration snippets that enable log replay are dead settings on any current cluster.

Reading the WAL, and the metrics that page you

The log is inspectable. WALPrettyPrinter dumps entries as readable records, which is the fastest way to answer "did this mutation ever reach the server" and the only way to see meta edits directly:

# dump a log; -j for JSON, -r/-w to filter by region or row
hbase wal /hbase/WALs/rs7.example.com,16020,1717430400123/rs7...1717430400500

# the shell side
hbase> roll_wal_writer 'rs7.example.com,16020,1717430400123'
hbase> alter 'events', DURABILITY => 'ASYNC_WAL'    # policy, not mechanism
hbase> status 'replication'                          # who is pinning oldWALs

Two of these are storage incidents, two are retention bookkeeping, and none of them are fixed by changing the durability level — the reflex worth unlearning. Sync latency is the write path's heartbeat, and it is the first number to put on a dashboard for any HBase cluster that takes writes.