Hive Storage Handlers, in depth: HBase, Kafka and JDBC tables, the handler contract and when to copy instead

Where a storage handler sits

The contract

A storage handler is a Java class implementing org.apache.hadoop.hive.ql.metadata.HiveStorageHandler. The core methods named in the Hive design documentation are getInputFormatClass(), getOutputFormatClass(), getSerDeClass(), getMetaHook() and configureTableJobProperties(TableDesc, Map). The first three replace what STORED AS and ROW FORMAT normally supply, which is why the DDL rule is that STORED BY excludes both. The job-properties method copies table settings (a topic name, a JDBC URL) into the job configuration so tasks can see them.

The meta hook, HiveMetaHook, runs around metastore DDL: preCreateTable, commitCreateTable, rollbackCreateTable, preDropTable, commitDropTable(table, deleteData) and rollbackDropTable. This is how creating a managed HBase-backed table in Hive also creates the HBase table. The design documentation is explicit that there is no two-phase commit between the metastore and the other system. If the metastore write fails after the HBase table was created, the rollback hook has to undo it, and if that also fails you have an orphan.

Handlers can also opt into predicate pushdown by implementing HiveStoragePredicateHandler, whose decomposePredicate splits a WHERE clause into the part the external system can evaluate and a residual Hive evaluates itself. Predicate pushdown in Hive covers the general mechanism; with handlers it decides whether a query reads a key range or the whole external table.

Native, non-native, managed, external

The design documentation distinguishes four kinds of table. Managed native and external native are ordinary file-backed tables. Managed non-native tables have a handler and Hive manages the external object's lifecycle through the meta hook, so dropping the table can drop the HBase table. External non-native tables have a handler but another system owns the data; dropping the Hive table only removes the definition.

Default to external for anything another team or service writes. The JDBC handler documentation requires an external table, and Kafka topics are almost always owned elsewhere. The same documentation's early limitations (no ALTER TABLE on non-native tables, no CREATE TABLE AS SELECT into one) describe the original design; some have been relaxed since, the Iceberg handler for instance supports CTAS, so check your release rather than assuming either way. Note also that the handler class name is stored in the metastore table parameters, so every engine and client reading that catalog needs the handler JAR on its classpath or the table cannot even be described.

HBase: rows by key

The HBase handler, org.apache.hadoop.hive.hbase.HBaseStorageHandler, maps Hive columns onto an HBase row key and column-family qualifiers through the hbase.columns.mapping SerDe property. Exactly one entry must be :key. An entry like cf:col maps one qualifier; cf: on its own maps a whole column family to a Hive MAP column. Suffix #b for binary storage (for numbers written by Java clients with Bytes.toBytes) or #s for string. hbase.table.name is optional when the HBase name matches.

CREATE EXTERNAL TABLE customer_profile (
  customer_id STRING,
  email       STRING,
  tier        STRING,
  lifetime_spend BIGINT,
  flags       MAP<STRING, STRING>
)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
  "hbase.columns.mapping" = ":key,p:email,p:tier,m:spend#b,f:"
)
TBLPROPERTIES ("hbase.table.name" = "crm:customer_profile");

-- Point lookup on the row key: check the plan before trusting it.
EXPLAIN SELECT email, tier FROM customer_profile WHERE customer_id = 'C-1042';

Two semantics surprise people. HBase stores one row per key, so inserting two rows with the same key keeps only one: inserts are upserts. And INSERT OVERWRITE does not delete existing rows; it only overwrites the keys it writes. Whether an equality or range on the key column becomes an HBase scan range depends on your Hive version and settings, so run EXPLAIN and look for the filter being pushed into the table scan. If it is not, a WHERE clause on a key reads the whole HBase table through region servers, which is slow and loads a cluster serving live traffic.

Kafka: a topic as a table

The Kafka handler, org.apache.hadoop.hive.kafka.KafkaStorageHandler, exposes a topic as a table. Two table properties are mandatory: kafka.topic and kafka.bootstrap.servers. The handler appends four metadata columns to whatever payload columns you declare: __key (binary), __partition (int), __offset (bigint) and __timestamp (bigint). The payload is decoded by a SerDe set in table properties; check the README for your release's default and options.

CREATE EXTERNAL TABLE orders_stream (
  order_id    BIGINT,
  customer_id STRING,
  amount_cents BIGINT,
  status      STRING
)
STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
TBLPROPERTIES (
  "kafka.topic" = "orders.v1",
  "kafka.bootstrap.servers" = "kafka-1:9092,kafka-2:9092"
);

-- Read one hour without scanning the topic: the handler pushes these down.
SELECT order_id, amount_cents, `__partition`, `__offset`
FROM orders_stream
WHERE `__timestamp` >= 1791043200000 AND `__timestamp` < 1791046800000;

The README documents pushdown for __timestamp filters and for __partition and __offset comparisons combined with AND and OR. Any other predicate means reading every retained message. For writes, kafka.write.semantic accepts AT_LEAST_ONCE (the default) and EXACTLY_ONCE. Remember that a topic is not a table: retention deletes old data, so the same query returns different rows tomorrow, and a compacted topic keeps only the latest value per key.

JDBC: another database as a table

The JDBC handler, org.apache.hive.storage.jdbc.JdbcStorageHandler, reads a table or query from MySQL, PostgreSQL, Oracle, Derby or DB2. It requires an external table and the properties hive.sql.database.type, hive.sql.jdbc.url, hive.sql.jdbc.driver, hive.sql.dbcp.username and a password, plus either hive.sql.table or hive.sql.query. Do not use the clear-text hive.sql.dbcp.password: it is stored in clear text in the metastore table parameters. Use hive.sql.dbcp.password.keystore and hive.sql.dbcp.password.key instead.

CREATE EXTERNAL TABLE crm_customers (
  customer_id STRING,
  country     STRING,
  segment     STRING
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
  "hive.sql.database.type" = "MYSQL",
  "hive.sql.jdbc.driver"   = "com.mysql.cj.jdbc.Driver",
  "hive.sql.jdbc.url"      = "jdbc:mysql://crm-replica:3306/crm",
  "hive.sql.dbcp.username" = "hive_reader",
  "hive.sql.dbcp.password.keystore" = "jceks://hdfs/secure/hive_reader.jceks",
  "hive.sql.dbcp.password.key" = "crm.password",
  "hive.sql.table" = "customers",
  "hive.sql.partitionColumn" = "customer_id",
  "hive.sql.numPartitions" = "8"
);

The documentation states that columns map by position, not by name, and that values that fail type conversion become NULL. Reordering columns in the source therefore silently shuffles data. Splitting reads across tasks uses hive.sql.partitionColumn, hive.sql.numPartitions and optionally hive.sql.lowerBound and hive.sql.upperBound; each split is a separate query, so eight splits are eight concurrent connections. Hive pushes filters, projections, joins, unions, aggregations and sorts to the database where it can. Writing through the JDBC handler is not supported according to the Apache documentation. Point it at a replica, never the primary.

Writing your own handler

Write your own handler only when a system has no existing one. The easiest start is to extend DefaultStorageHandler and override what you need. The skeleton below uses only the methods named above; the InputFormat and RecordReader do the real work of turning the external system's data into splits and rows.

public class LedgerStorageHandler extends DefaultStorageHandler {
  @Override public Class<? extends InputFormat> getInputFormatClass() { return LedgerInputFormat.class; }
  @Override public Class<? extends OutputFormat> getOutputFormatClass() { return LedgerOutputFormat.class; }
  @Override public Class<? extends AbstractSerDe> getSerDeClass() { return LedgerSerDe.class; }

  @Override public void configureTableJobProperties(TableDesc desc, Map<String, String> jobProps) {
    // Copy only what tasks need; never copy secrets into the job configuration.
    jobProps.put("ledger.endpoint", desc.getProperties().getProperty("ledger.endpoint"));
    jobProps.put("ledger.book", desc.getProperties().getProperty("ledger.book"));
  }

  @Override public HiveMetaHook getMetaHook() { return new LedgerMetaHook(); }
}

class LedgerMetaHook implements HiveMetaHook {
  public void preCreateTable(Table t) throws MetaException {
    if (!MetaStoreUtils.isExternalTable(t)) throw new MetaException("ledger tables must be EXTERNAL");
  }
  public void commitCreateTable(Table t) {}
  public void rollbackCreateTable(Table t) {}
  public void preDropTable(Table t) {}
  public void commitDropTable(Table t, boolean deleteData) {}   // never delete the ledger
  public void rollbackDropTable(Table t) {}
}

Package the handler with its dependencies, add it to the classpath of HiveServer2 and every task container (or with ADD JAR for testing), and run DDL plus a scan, a filtered scan and a drop in a test environment. The SerDe side follows the same contract as any other; see Hive SerDes.

Worked example: hourly revenue from Kafka and MySQL

Worked example: finance wants hourly revenue by customer segment, using orders on a Kafka topic and segments in a MySQL CRM. With the two tables above, the job is one INSERT into an ordinary ORC table partitioned by hour:

INSERT INTO TABLE revenue_by_segment PARTITION (hr = '2026-10-03-16')  -- 16:00-17:00 UTC
SELECT c.segment, COUNT(*) AS orders, SUM(o.amount_cents) AS revenue_cents
FROM orders_stream o
JOIN crm_customers c ON o.customer_id = c.customer_id
WHERE o.`__timestamp` >= 1791043200000 AND o.`__timestamp` < 1791046800000
  AND o.status = 'PAID'
GROUP BY c.segment;

The timestamp predicate becomes per-partition offset ranges in Kafka, so only one hour is read. The CRM side is read in eight splits from the replica. The status filter cannot be pushed into Kafka and runs in Hive. Land the result in a native table, not a handler table: re-running the hour is an idempotent INSERT OVERWRITE of one partition, and analysts query fast ORC files instead of hitting Kafka and MySQL every time. Watch one trap: Kafka message timestamps can be producer-set, so a late or skewed producer puts orders in the wrong hour. Run the hour after a grace period and reconcile counts against the source.

Failure modes

• Missing JAR. A client without the handler JAR cannot read the table definition. Fix: deploy handler JARs everywhere the catalog is used, including Impala, Spark and BI gateways, or keep handler tables in a separate database.
• Accidental drop. A managed HBase-backed table is dropped and the HBase table goes with it. Fix: EXTERNAL for anything shared.
• Full scans. A predicate that is not pushed down reads all of HBase or all of a topic's retention. Fix: EXPLAIN every recurring query.
• Positional JDBC mapping. A column added in the middle of the source table shifts values into the wrong Hive columns. Fix: use hive.sql.query with an explicit column list.
• Source overload. A large join opens many connections to a production database. Fix: replicas, a small hive.sql.numPartitions, and a dedicated database user with limits.
• Leaked passwords. Clear-text passwords in TBLPROPERTIES. Fix: keystore properties.
• Engine gaps. Impala can query HBase tables defined in Hive and has its own Kudu and Iceberg support (see Impala over HBase and Impala and Kudu), but it does not run arbitrary Hive storage handlers; check before promising a table to Impala users.

Trade-offs

A handler table is the right tool for exploration, small lookups and periodic extracts: no copy, always current. It is the wrong tool as the main source for dashboards, because every query lands on a system built for something else, statistics are poor, and results change as the source changes. The common pattern is handler table as the edge, native or Iceberg table as the store: read through the handler on a schedule, write to a file-backed table, query that. Iceberg itself is a handler, org.apache.iceberg.mr.hive.HiveIcebergStorageHandler, covered in Iceberg tables in Hive; it differs because the data is still files Hive can plan over efficiently.

What to do next

1. List every table with a handler: query the metastore's table parameters for the storage handler key.
2. Make each shared handler table EXTERNAL and confirm what DROP does in a test environment.
3. Move JDBC passwords to a keystore and repoint JDBC tables at replicas.
4. Run EXPLAIN on every scheduled query over a handler table and confirm pushdown.
5. Replace dashboard reads of handler tables with scheduled loads into native or Iceberg tables.
6. Deploy handler JARs to every engine that shares the catalog, or isolate those tables.