HBase + Ozone Integration, in depth: running HBase on Apache Ozone

Why run HBase on Ozone

The motivation is mostly about the NameNode. HDFS keeps every file and block in one NameNode's heap, so clusters with hundreds of millions of files need federation or careful compaction of small files. Ozone splits the job: the Ozone Manager (OM) holds the namespace (volumes, buckets, keys) in RocksDB and replicates it with Apache Ratis, a Raft implementation; the Storage Container Manager (SCM) manages containers, which are large groups of blocks, so block reports are per container rather than per block; and Datanodes store container data. One cluster can serve S3 clients and Hadoop clients from the same buckets.

For an organisation moving off HDFS, being able to run HBase, Hive, Spark and S3 workloads on one storage layer means one capacity pool, one security model and one upgrade path. The Apache Ozone deep dive covers Ozone itself; this page is about the HBase side.

It is not automatically the right move. A small HBase cluster on a healthy HDFS gains little and takes on a younger integration, a second consensus system in the write path and a list of unsupported features. The case is strongest when HDFS namespace pressure is the real constraint, when the organisation is already standardising on Ozone for other workloads, or when HDFS is being retired and HBase is the last tenant holding it in place.

What HBase needs from a filesystem

It helps to list exactly what HBase assumes, because each item maps to a specific Ozone feature.

Historically HBase called these through HDFS-specific classes such as DistributedFileSystem.recoverLease. HADOOP-18671 added recoverLease(), setSafeMode() and isFileClosed() as optional capabilities on the generic Hadoop FileSystem layer, and HBase work under HBASE-27740 uses them through capability checks, so HBase does not need Ozone classes at compile time. On the Ozone side, HDDS-7593 added hsync and lease recovery. Ozone 2 shipped them, explicitly to enable HBase and Solr log workloads.

HBase also refuses to start if its WAL output stream does not advertise hflush and hsync capabilities, unless hbase.unsafe.stream.capability.enforce is set to false. Do not set it to false to get past a startup error on Ozone: that error is telling you hsync is disabled, and running without it means acknowledged writes can vanish after a crash. See the HBase WAL for the write path in detail.

How Ozone provides it

Three Ozone choices make the guarantees work. The ofs:// scheme is a rooted filesystem view in which volumes and buckets appear as top-level directories, so ofs://ozone1/hbasevol/hbasebucket/hbase is a normal Hadoop path; the older o3fs:// scheme is not supported for HBase. File System Optimized (FSO) buckets store a real directory tree in OM, so renaming or deleting a directory is a metadata update rather than a rewrite of every key under a prefix, and renames are atomic; Object Store (OBS) buckets lack this and are not supported. Ratis replication writes each block through a three-node Raft pipeline, which is what lets an hsync return only once data is durable on a quorum; erasure-coded buckets are not supported for HBase, much as erasure coding under HBase on HDFS has its own WAL restrictions.

Ozone also added write-path optimisations aimed at HBase's pattern of many small synced appends: piggybacking the block-metadata commit (PutBlock) on data writes, and an incremental chunk list so each sync does not resend the whole block's metadata. These, together with hsync and lease recovery, are gated behind explicit opt-in flags.

Configuration

On the Ozone cluster and in the client configuration HBase loads, set the following. The two enhancement switches are the server and client halves of the HBase opt-in; the Ozone project describes the gated features as requiring this extra switch.

<!-- ozone-site.xml (OM and clients) -->
<property><name>ozone.fs.hsync.enabled</name><value>true</value></property>
<property><name>ozone.hbase.enhancements.allowed</name><value>true</value></property>
<property><name>ozone.client.hbase.enhancements.allowed</name><value>true</value></property>
<property><name>ozone.client.stream.putblock.piggybacking</name><value>true</value></property>
<property><name>ozone.client.incremental.chunk.list</name><value>true</value></property>

Create the volume and an FSO bucket with Ratis replication, then point HBase at it.

ozone sh volume create /hbasevol
ozone sh bucket create /hbasevol/hbasebucket --layout FILE_SYSTEM_OPTIMIZED \
    --type RATIS --replication THREE

<!-- hbase-site.xml -->
<property><name>hbase.rootdir</name>
  <value>ofs://ozone1/hbasevol/hbasebucket/hbase</value></property>

The Ozone filesystem client jar must be on HBase's classpath, and HBase must be built against a Hadoop release that includes the HADOOP-18671 interfaces; distribution builds differ, so check your vendor's matrix. If Ozone runs with Kerberos, HBase's principal needs ACLs on the volume and bucket. Before loading data, run a short smoke test: create a table, write, kill -9 a RegionServer and confirm every acknowledged row is readable afterwards.

Worked example: a write and a crash

Walk through one write and one crash. A client sends a batch of 100 puts to region R on RegionServer RS1.

1. RS1 appends the batch to its WAL file, an open Ozone key under ofs://ozone1/hbasevol/hbasebucket/hbase/WALs/rs1,.... The append goes into the Ozone client's buffer.
2. RS1 calls hsync. The client pushes buffered chunks to the three Datanodes of the block's Ratis pipeline and commits block metadata; with hsync enabled, OM records the open key's current length so it is visible and recoverable. Only then does RS1 apply the puts to the MemStore and acknowledge the client.
3. RS1 dies. Its ZooKeeper session expires and the HMaster schedules server crash procedures.
4. Before splitting the WAL, the master calls recoverLease on each of RS1's WAL files. Ozone marks the open key as under recovery, determines the final committed length from the Datanodes, and closes it. isFileClosed now returns true.
5. The WAL is split per region and replayed; R reopens on RS2 with all 100 puts present, because every acknowledged edit was synced to a Ratis quorum before the acknowledgement.

Two numbers drive the experience. WAL sync latency sets write latency; on HDFS it is a pipeline of Datanode writes, on Ozone it is a Ratis round trip plus metadata, so measure p99 sync time under your write rate before cut-over. Recovery time is dominated by lease recovery plus WAL split; test it with realistic WAL sizes. Ozone also has ozone.om.lease.hard.limit (seven days by default) governing how long hsync'd files can stay open before the open-key cleanup service acts on them, which matters if a RegionServer is wedged rather than dead.

What is not supported

The Ozone integration guide lists these as not currently supported:

• HBase MOB (medium objects), favored nodes, hedged reads, storage policies and short-circuit reads.
• Ozone snapshots, erasure-coded buckets and OBS buckets.
• Phoenix UDFs.

Read the list carefully. Ozone snapshots are an Ozone bucket feature; HBase's own table snapshots are a different mechanism built from HFile links and manifests, and the guide does not list them as unsupported, but test them on your build before relying on them for backup. Losing short-circuit reads matters more than it sounds: on HDFS a RegionServer co-located with a Datanode reads HFile blocks straight from local disk, and that locality is a large part of HBase's random-read latency. On Ozone reads go through the client protocol, so a bigger BucketCache is the usual compensation. Favored nodes and storage policies were how operators pinned regions to hosts and put WALs on SSD; without them you plan placement at the Ozone pipeline level instead.

Failure modes

• hsync disabled. HBase refuses to start with a stream capability error, or worse, someone disables enforcement and acknowledged writes are lost on crash. Verify the flags on both OM and clients.
• Wrong bucket layout. An OBS or legacy bucket makes renames non-atomic and slow; flushes and compactions misbehave. The bucket must be created as FSO and cannot be converted in place.
• EC bucket by default. If the cluster default replication is EC, a bucket created without --type RATIS silently gets it.
• Slow lease recovery. Region reassignment stalls while the master waits for WALs to close; watch server crash procedure duration.
• OM as bottleneck. Every WAL roll, flush, compaction and split is OM metadata traffic. A big compaction storm shows up as OM RPC latency, which then shows up as HBase write latency. Size OM handlers and RocksDB disks for it.
• Version skew. HBase built against a Hadoop without the generic lease-recovery interfaces cannot recover WALs on Ozone even though writes work, which surfaces only on the first crash. That is why the smoke test kills a server.

Operational guidance and trade-offs

Several designs reduce risk. The most conservative is a hybrid: hbase.wal.dir lets the WAL live on a different filesystem from hbase.rootdir, so you can keep WALs on a small HDFS or fast storage and put HFiles, which are written once and read many times, on Ozone. That removes the hsync path from the critical latency budget while still moving the bulk of capacity. The opposite trade-off, everything on Ozone, removes HDFS entirely and is the end state the integration aims at.

Migrate table by table with snapshots or bulk loads rather than copying the root directory: take an HBase snapshot on the HDFS cluster, ExportSnapshot it to the Ozone-backed cluster, clone it, then replay the tail through replication. Bulk load is the other route for tables you can rebuild.

Plan capacity on both sides. Ratis THREE replication stores three full copies, the same raw overhead as HDFS with replication three, so there is no capacity saving from the move itself; savings come from consolidating clusters or from putting colder, non-HBase data on erasure-coded buckets alongside. Compactions rewrite HFiles continuously, so Datanode disk throughput and OM write capacity must absorb that background load as well as client traffic.

Monitor both systems as one: HBase WAL sync latency, flush queue, compaction queue and server crash procedure time next to OM RPC latency, OM RocksDB size, SCM pipeline health and Datanode disk latency. Most HBase-on-Ozone incidents present in HBase metrics but originate in Ozone metadata or pipeline state.

What to do next

1. Confirm your Ozone release ships hsync and lease recovery (Ozone 2 or a vendor build that backports them) and that your HBase build was compiled against Hadoop with the generic lease-recovery interfaces.
2. Set the five Ozone flags on OM and clients; create an FSO, Ratis THREE bucket.
3. Point hbase.rootdir at an ofs:// path and leave stream capability enforcement on.
4. Run the crash test: write, kill -9 a RegionServer, verify every acknowledged row and time the recovery.
5. Benchmark p99 WAL sync latency and random-read latency against your HDFS baseline; size BucketCache to compensate for lost short-circuit reads.
6. Decide between a hybrid layout (WAL on HDFS) and full Ozone, and check that no table uses MOB or Phoenix UDFs.
7. Migrate one non-critical table via ExportSnapshot and run it for a release cycle before moving more.