HBase Java API, in depth: Put, Get, Scan and Delete semantics, atomic operations, batch results and Admin, with a tested data-access layer

The data model as the API sees it

An HBase table maps a row key to column families; each family holds any number of qualifiers; each (row, family, qualifier) holds one or more versions distinguished by a timestamp. The unit the API returns is a Cell: row, family, qualifier, timestamp, type (a put or one of the delete markers) and value. Every one of those, except timestamp and type, is a raw byte[]. HBase has no column types and no schema beyond the family names.

So the first API you use is Bytes, the utility that converts Java values to and from byte arrays: Bytes.toBytes(long) gives eight big-endian bytes, Bytes.toBytes(String) gives UTF-8. Encoding is a contract between writers and readers, and it determines sort order: rows are sorted by unsigned lexicographic comparison of their key bytes, so a big-endian non-negative long sorts numerically, while the string "10" sorts before "9". Decide encodings once, in one class.

How an operation reaches a region

A Table is a cheap, non-thread-safe handle; obtain one per unit of work with connection.getTable(name) and close it with try-with-resources. Operation objects such as Put are plain builders that touch the network only when passed to the table, which finds each row's region from its cache or hbase:meta.

Creating tables with Admin

TableName EVENTS = TableName.valueOf("user_events");
byte[] D = Bytes.toBytes("d");

try (Admin admin = connection.getAdmin()) {
    TableDescriptor desc = TableDescriptorBuilder.newBuilder(EVENTS)
        .setColumnFamily(ColumnFamilyDescriptorBuilder.newBuilder(D)
            .setMaxVersions(1)
            .setTimeToLive(90 * 24 * 3600)          // seconds; expired cells vanish at compaction
            .setBloomFilterType(BloomType.ROW)
            .setCompressionType(Compression.Algorithm.SNAPPY)  // codec must exist on servers
            .build())
        .build();
    // 16 salt buckets -> 15 split points, so writes spread from the first second
    byte[][] splits = new byte[15][];
    for (int i = 1; i < 16; i++) splits[i - 1] = new byte[] { (byte) i };
    if (!admin.tableExists(EVENTS)) admin.createTable(desc, splits);
}

Table creation belongs in deployment code, not at application start-up, but writing it in Java makes the schema reviewable. The descriptor builders are immutable: you configure, then build(). Here one short family name, d, keeps every cell's stored key small; one version is enough for an event log; a time-to-live of 90 days lets compaction discard old events with no delete traffic; a row Bloom filter speeds point reads. Pre-splitting matters most: a new table starts as one region on one server, and a burst of writes all lands there. Fifteen split points on the first key byte give sixteen regions from the first write.

Writing: Put and its guarantees

A Put holds one row's worth of cells. Each addColumn(family, qualifier, value) adds a cell; an overload takes an explicit timestamp, and without one the RegionServer assigns the current time. A put to a single row is atomic: all its cells become visible together or none do. There is no insert-versus-update distinction. Writing the same row, column and timestamp again overwrites; writing with a new timestamp adds a version, kept up to the family's maximum.

Because a put with explicit values is idempotent, retrying it after a timeout is safe: the worst case is writing the same cell twice. That property is worth preserving in your design. Durability per mutation can be set with setDurability; the default follows the table, and the trade-offs between sync and async write-ahead logging are covered in WAL durability.

Deleting: three methods that look alike

Deletes are the most misread part of the API. HBase never removes data in place; a Delete writes a tombstone marker that hides older cells until a major compaction physically removes both. The methods differ in which cells the marker covers:

• new Delete(row) with nothing added: every column in every family of the row.
• addFamily(family): all cells in that family.
• addColumns(family, qualifier), plural: all versions of that column.
• addColumn(family, qualifier), singular: only the latest version. If the column keeps several versions, the previous one reappears, which surprises almost everyone the first time.

Heavy delete traffic leaves tombstones that slow scans until compaction, and a put timestamped older than an existing marker stays hidden until that marker is compacted away.

Reading: Get, Result and Cell

A Get fetches one row. Narrow it before sending: addFamily or addColumn limit what the server reads and returns; readVersions(n) asks for up to n versions; setTimeRange(min, max) restricts timestamps; setFilter applies server-side filters. table.get(List<Get>) groups many gets into per-server calls.

The answer is a Result. A missing row is not null; it is an empty result, so test result.isEmpty(). getValue(family, qualifier) returns the newest value or null; rawCells() returns every cell, sorted by family, qualifier and then newest timestamp first. When iterating cells, copy fields out with CellUtil.cloneQualifier and CellUtil.cloneValue, because a cell's accessors return offsets into a shared backing array, not standalone arrays.

Worked example: a user event store

A product team wants to record user events (clicks, purchases, logins) and show each user's most recent events. The access pattern drives the row key: one byte of salt so writes spread across the sixteen pre-split regions, then the user ID so a user's events are contiguous, then a reversed timestamp so the newest event sorts first. The data-access class:

public final class UserEventDao {
    private static final byte[] D = Bytes.toBytes("d");
    private static final byte[] TYPE = Bytes.toBytes("type");
    private static final byte[] BODY = Bytes.toBytes("body");
    private final Connection conn;                       // shared, created once at startup

    public UserEventDao(Connection conn) { this.conn = conn; }

    // row = bucket(1) + userId(8) + (Long.MAX_VALUE - tsMillis)(8): newest events sort first
    static byte[] rowKey(long userId, long tsMillis) {
        byte bucket = (byte) Math.floorMod(Long.hashCode(userId), 16);
        return Bytes.add(new byte[] { bucket }, Bytes.toBytes(userId),
                         Bytes.toBytes(Long.MAX_VALUE - tsMillis));
    }

    static byte[] userPrefix(long userId) {
        return Arrays.copyOf(rowKey(userId, 0L), 9);     // bucket + userId
    }

    public void record(long userId, long tsMillis, String type, byte[] body) throws IOException {
        Put put = new Put(rowKey(userId, tsMillis))
            .addColumn(D, TYPE, Bytes.toBytes(type))
            .addColumn(D, BODY, body);
        try (Table t = conn.getTable(EVENTS)) { t.put(put); }   // idempotent: safe to retry
    }

    public List<UserEvent> latest(long userId, int limit) throws IOException {
        byte[] start = userPrefix(userId);
        Scan scan = new Scan()
            .withStartRow(start)
            .withStopRow(Bytes.unsignedCopyAndIncrement(start))   // exclusive end of the prefix
            .addFamily(D)
            .setLimit(limit)                                      // stop server-side after N rows
            .setCaching(Math.min(limit, 100));
        List<UserEvent> out = new ArrayList<>();
        try (Table t = conn.getTable(EVENTS); ResultScanner rs = t.getScanner(scan)) {
            for (Result r : rs) {
                long ts = Long.MAX_VALUE - Bytes.toLong(r.getRow(), 9);
                out.add(new UserEvent(userId, ts,
                        Bytes.toString(r.getValue(D, TYPE)), r.getValue(D, BODY)));
            }
        }
        return out;
    }
}

Two details carry the design. The scan's stop row is the prefix incremented by one, computed with Bytes.unsignedCopyAndIncrement: scans are half-open ranges, and without a correct stop row the scan runs into the next user's rows, the classic source of rows from the wrong user. And setLimit tells the server to stop after the requested number of rows, while setCaching sets how many rows each RPC carries; a large caching value with no limit fetches far more than the page needs. More scan tuning is in HBase scans.

Atomic operations: counters, compare-and-set, multi-cell row changes

byte[] row = Bytes.toBytes("user:42");
byte[] C = Bytes.toBytes("c");

try (Table t = conn.getTable(COUNTERS)) {
    // 1. Counter: stored as an 8-byte long; returns the new value
    long views = t.incrementColumnValue(row, C, Bytes.toBytes("views"), 1L);

    // 2. Compare-and-set: claim a coupon only if nobody has claimed it (HBase 2.4+ API)
    CheckAndMutate claim = CheckAndMutate.newBuilder(Bytes.toBytes("coupon:SPRING"))
        .ifNotExists(C, Bytes.toBytes("owner"))
        .build(new Put(Bytes.toBytes("coupon:SPRING"))
            .addColumn(C, Bytes.toBytes("owner"), Bytes.toBytes(42L)));
    boolean won = t.checkAndMutate(claim).isSuccess();

    // 3. Several changes to ONE row, applied atomically
    RowMutations rm = new RowMutations(row)
        .add(new Put(row).addColumn(C, Bytes.toBytes("status"), Bytes.toBytes("active")))
        .add(new Delete(row).addColumns(C, Bytes.toBytes("pending_since")));
    t.mutateRow(rm);
}

HBase is atomic per row and offers no transactions across rows. Within a row, three tools cover most needs. Increment (or the incrementColumnValue shortcut) adds to a counter stored as an eight-byte long and returns the new value; writing a counter column with a string or int encoding first makes later increments fail. Append appends bytes to an existing value. CheckAndMutate, the 2.4 API that replaced the older checkAndPut-style builders, applies a mutation only if a condition on one column holds, which is enough for claims, leases and optimistic concurrency. RowMutations applies several puts and deletes to one row as a unit.

Unlike a put, an increment or append changes state relative to the current value, so a blind retry can apply it twice. HBase's own client attaches nonces to these operations so the server can recognise a duplicate of a call it has already applied, but a retry in your application code after an ambiguous failure is a new call. If double counting matters, guard with a compare-and-set on an operation ID, or write idempotent per-event cells.

Batch calls and partial failure

List<Row> actions = new ArrayList<>();
for (Event e : events) actions.add(toPut(e));
Object[] results = new Object[actions.size()];
try (Table t = conn.getTable(EVENTS)) {
    t.batch(actions, results);
} catch (RetriesExhaustedWithDetailsException ex) {
    // some actions failed after retries; results[] still tells you which
    for (int i = 0; i < ex.getNumExceptions(); i++) {
        log.warn("row {} failed on {}: {}", Bytes.toStringBinary(ex.getRow(i).getRow()),
                 ex.getHostnamePort(i), ex.getCause(i).toString());
    }
}
for (int i = 0; i < results.length; i++) {
    if (results[i] == null || results[i] instanceof Throwable) retryQueue.add(actions.get(i));
}

table.batch sends a mixed list of puts, deletes, gets and increments, grouped per RegionServer, and fills a parallel results array: a Result for each success and a Throwable (or null) for each action that failed after the client's retries. Batches are not atomic; any subset may succeed. When failures remain, the call also throws RetriesExhaustedWithDetailsException, which lists each failed row, the server and the cause. Handle both: requeue exactly the failed actions, and never retry the whole batch if it contains non-idempotent increments. For sustained ingest, the buffered mutator described in the client libraries guide is usually the better tool, and write performance covers sizing.

Testing against a real cluster in-process

class UserEventDaoIT {
    static HBaseTestingUtility util = new HBaseTestingUtility();   // HBaseTestingUtil in 3.x

    @BeforeAll static void up() throws Exception { util.startMiniCluster(); /* create table */ }
    @AfterAll  static void down() throws Exception { util.shutdownMiniCluster(); }

    @Test void newestFirstBoundedAndLimited() throws Exception {
        UserEventDao dao = new UserEventDao(util.getConnection());
        for (long ts = 1; ts <= 5; ts++) dao.record(7L, ts, "click", new byte[0]);
        dao.record(23L, 3L, "click", new byte[0]);   // same salt bucket as user 7, next in key order
        assertEquals(List.of(5L, 4L, 3L, 2L, 1L),
                dao.latest(7L, 10).stream().map(UserEvent::ts).toList());   // stop row holds
        assertEquals(List.of(5L, 4L, 3L),
                dao.latest(7L, 3).stream().map(UserEvent::ts).toList());    // limit holds
    }
}

The mini-cluster from the hbase-testing-util artifact starts ZooKeeper, a master and a RegionServer inside the JVM, so tests exercise real row-key ordering, real scans and real delete semantics rather than a mock that agrees with your assumptions. The class is HBaseTestingUtility in 2.x and was renamed HBaseTestingUtil in 3.x. Start it once per test class, and include a neighbouring user in fixtures so a wrong stop row fails the test.

Failure modes

• Table handle shared across threads. Table is not thread-safe; share the Connection and obtain a table per task.
• Scanner never closed. An unclosed ResultScanner holds server-side resources until its lease expires; a slow consumer that pauses longer than the lease sees a lease-expired error. Use try-with-resources and keep per-row work short.
• Wrong stop row. Prefix scans without an exclusive incremented stop row return other entities' data.
• Singular addColumn on a versioned column. Removes only the newest version and resurrects the previous one.
• Mixed encodings. One service writes a counter as a string, another increments it; or one writes ints and another reads longs. Centralise encoding.

Trade-offs

What to do next

1. Write one encoding class per table that owns row-key construction and every field's byte encoding, and forbid ad-hoc Bytes calls elsewhere.
2. Audit every Delete in your codebase for singular addColumn on families that keep more than one version.
3. Give every prefix scan an explicit exclusive stop row, a limit and a caching value sized to the page.
4. Find every Increment and Append, and decide whether an application retry could double-apply it; if so, switch to compare-and-set with an operation ID or idempotent per-event cells.
5. Inspect batch results and RetriesExhaustedWithDetailsException in every batch call, and requeue only the failed actions.
6. Add a mini-cluster integration test per data-access class that includes neighbouring rows in its fixtures.