HBase Security, in depth: Kerberos, ZooKeeper SASL, delegation tokens, wire encryption and Ranger authorization

One request through a secure cluster

Follow a Get from a Java client. The client first needs Kerberos credentials: a ticket-granting ticket from kinit or a login from a keytab. It connects to ZooKeeper to find the hbase:meta location, authenticating with SASL. It then opens an RPC connection to the RegionServer that hosts the row, and SASL GSSAPI authenticates that connection with a service ticket for hbase/rs-host@REALM. From that point the server knows the caller's principal.

Before the read executes, the authorization coprocessor checks whether that principal holds READ on the table, column family or column. If it does not, the client gets an AccessDeniedException. If it does, the read proceeds, the decision is audited, and the RegionServer reads HFiles from HDFS as the hbase service principal, not as the end user. That last point shapes everything: HDFS sees only HBase, so per-user permissions must be enforced inside HBase. File permissions on /hbase only stop people from bypassing HBase and reading HFiles directly.

Kerberos for Masters, RegionServers and clients

Each Master and RegionServer runs as a service principal, normally hbase/_HOST@REALM, where _HOST is replaced by the machine's fully qualified host name at startup. Each host needs a keytab holding its own key. The core server settings in hbase-site.xml:

<property><name>hbase.security.authentication</name><value>kerberos</value></property>
<property><name>hbase.security.authorization</name><value>true</value></property>
<property><name>hbase.master.kerberos.principal</name><value>hbase/_HOST@EXAMPLE.COM</value></property>
<property><name>hbase.master.keytab.file</name><value>/etc/security/keytabs/hbase.service.keytab</value></property>
<property><name>hbase.regionserver.kerberos.principal</name><value>hbase/_HOST@EXAMPLE.COM</value></property>
<property><name>hbase.regionserver.keytab.file</name><value>/etc/security/keytabs/hbase.service.keytab</value></property>
<property><name>hbase.coprocessor.region.classes</name>
  <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value></property>
<property><name>hbase.coprocessor.master.classes</name>
  <value>org.apache.hadoop.hbase.security.access.AccessController</value></property>
<property><name>hbase.superuser</name><value>hbase,@hbase-admins</value></property>

Clients need hbase.security.authentication set to kerberos and the server principal patterns, so they know which service ticket to request. _HOST depends on DNS: forward and reverse lookups must agree, or the client asks for a ticket for the wrong principal. A long-running service logs in from its own keytab instead of relying on a user's ticket cache:

Configuration conf = HBaseConfiguration.create();
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("svc-fraud@EXAMPLE.COM",
    "/etc/security/keytabs/svc-fraud.keytab");
try (Connection conn = ConnectionFactory.createConnection(conf);
     Table t = conn.getTable(TableName.valueOf("fraud:scores"))) {
  Result r = t.get(new Get(Bytes.toBytes("acct#000123")));
}
// In long-running processes, call this periodically (for example before each batch):
UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab();

Securing ZooKeeper

HBase stores cluster state in ZooKeeper: the active Master, the hbase:meta location, region-in-transition records and replication peers. If ZooKeeper is open, anyone can delete those znodes and take the cluster down. Configure the ZooKeeper servers to accept SASL, with authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider and kerberos.removeHostFromPrincipal=true plus kerberos.removeRealmFromPrincipal=true, so every HBase host authenticates as plain hbase. Give HBase a JAAS file with a Client section that uses its keytab. When HBase runs secure, it creates its znodes with ACLs that restrict writes to the HBase principal.

One trap: znodes created before security was enabled keep their old open ACLs. When you secure an existing cluster, check the ACLs under /hbase with getAcl afterwards, and fix or recreate any that are still world-writable during a maintenance window.

Delegation tokens for batch jobs

MapReduce and Spark tasks run on worker nodes that have no user keytab. They use delegation tokens. The TokenProvider coprocessor (in the region coprocessor list above) issues an HBase authentication token to a Kerberos-authenticated client. The job driver obtains it before submitting work, the framework ships it with the job, and tasks authenticate with the token instead of Kerberos. For MapReduce, TableMapReduceUtil.initCredentials(job) does this; Spark obtains HBase tokens at submission when HBase is on its classpath and configured for Kerberos.

Tokens expire. A job that outlives the token's lifetime fails partway through with authentication errors even though nothing changed. Long-running streaming jobs need a keytab and principal passed to the framework so it can re-obtain tokens, rather than a token from the user's session. Bulk loads also need care: secure bulk load is built into HBase 2, and it moves the job's HFiles through a staging directory owned by HBase, so the job user never needs write access to /hbase.

Protecting data on the wire

Kerberos proves identity but does not by itself encrypt the data on the wire. HBase offers two ways to protect RPC traffic.

The older is SASL quality of protection, set by hbase.rpc.protection: authentication (identity only, the default), integrity (adds checksums) or privacy (encrypts). Clients and servers must agree. privacy is simple to turn on but costs noticeable CPU on busy RegionServers, so load-test it.

HBase 2.6.0 added native TLS for RPC, built on Netty, so it works only with the Netty RPC client and server. Servers set hbase.server.netty.tls.enabled and a keystore (hbase.rpc.tls.keystore.location, hbase.rpc.tls.keystore.password, hbase.rpc.tls.keystore.type); clients set hbase.client.netty.tls.enabled and a truststore (hbase.rpc.tls.truststore.*). The documented rolling enable uses hbase.server.netty.tls.supportplaintext so old and new clients overlap:

1. Enable server-side TLS with plaintext still allowed on the Master; restart it.
2. Enable server and client TLS, plaintext allowed, on RegionServers; rolling restart.
3. Enable client-side TLS on every client.
4. Enable client-side TLS on the Master and turn off plaintext there; restart.
5. Turn off plaintext on RegionServers; rolling restart.

TLS covers RPC only, not the web UIs, which need their own HTTPS settings. Data at rest is a separate decision: HDFS transparent encryption zones under /hbase, or HBase's per-column-family encryption, which also covers the WAL when configured.

Authorization: AccessController or Ranger

HBase ships its own authorization coprocessor, AccessController. Permissions are R (read), W (write), X (execute coprocessor endpoints), C (create, alter and drop tables) and A (admin). They are granted at global, namespace, table, column family or column scope from the shell, for example grant 'svc-fraud', 'RW', '@fraud' for a namespace, and stored in the hbase:acl table. It needs no extra service and works offline. Its weaknesses are management at scale: grants live inside each cluster, there is no central UI, and audit is a log line per decision.

hbase> whoami                                  # which principal and groups the server sees
hbase> grant 'svc-fraud', 'RW', '@fraud'       # namespace scope
hbase> grant '@fraud-analysts', 'R', 'fraud:scores', 's'   # group, table, family
hbase> user_permission '@fraud'                # list grants on the namespace
hbase> revoke 'svc-fraud', '@fraud'

Run whoami first whenever a permission looks wrong. It shows the principal and groups the server resolved for you, and most surprises are group mapping, not grants: the server resolves groups on its side, through its own Hadoop group mapping, not from what the client believes.

Apache Ranger replaces it with a central policy service. In both hbase.coprocessor.master.classes and hbase.coprocessor.region.classes, replace AccessController with org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor, keeping TokenProvider in the region list. Run one authorizer, not both. Policies match table, column family and column (wildcards allowed, such as fraud:*), grant read, write, create, admin and execute to users and groups, and can be time-bound or denied explicitly. The plugin pulls policies from Ranger Admin on an interval and caches them on local disk, so a Ranger Admin outage freezes policy changes but does not stop data access. Every decision goes to Ranger's audit store, typically Solr for search and HDFS for retention. Ranger Policy Deep Dive explains how policies are evaluated; HBase Multi-Tenancy covers how to lay out namespaces and grants per tenant.

Thrift and REST gateways

Thrift and REST gateways let non-Java clients reach HBase, which creates a trap. The gateway authenticates to HBase as its own principal, so without impersonation every request runs with the gateway's permissions and the audit log shows only the gateway. Enable impersonation with hbase.thrift.support.proxyuser or hbase.rest.support.proxyuser, and authorise the gateway principal to impersonate only the users and hosts you intend, using the hadoop.proxyuser.<gateway-user>.hosts and .groups entries in core-site.xml. The REST server authenticates callers with SPNEGO; for Thrift, hbase.thrift.security.qop sets the SASL protection level. HBase Thrift and REST Gateways in Depth covers the gateways themselves.

Worked example: onboarding a fraud team

A fraud team needs a service that writes scores, a nightly Spark job that reads transactions, and two analysts who read scores ad hoc. The cluster uses Ranger.

1. Create principals svc-fraud@EXAMPLE.COM and spark-fraud@EXAMPLE.COM; export keytabs readable only by the service accounts on their hosts.
2. Create namespace fraud as an HBase admin, and an AD group fraud-analysts.
3. Ranger policy 1: fraud:scores, all families, read and write for svc-fraud.
4. Ranger policy 2: default:transactions, family t, read for spark-fraud; the Spark job is submitted with that principal and keytab so tokens can be renewed.
5. Ranger policy 3: fraud:scores, read for group fraud-analysts, with a deny on column s:raw_features.
6. Verify as each identity: kinit, then hbase shell with get, put and scan; confirm one allowed and one denied action per identity in the Ranger audit.

The deny on raw_features shows why column-level policies cost something. When a family has column-level rules, scans must check access at the cell level, which is slower than a family-wide decision. Keep sensitive columns in their own family so the rule can be family-wide.

Failure modes

• Clock skew. Kerberos rejects tickets when clocks differ by more than the allowed skew (five minutes by default). Run NTP or chrony everywhere and alert on drift.
• Keytab rotated, process not restarted. A new key version makes old tickets fail with checksum or pre-authentication errors. Rotate keytabs with a rolling restart plan.
• Expired tickets in long-running clients. Processes that logged in once fail after the ticket lifetime. Use keytab login plus periodic checkTGTAndReloginFromKeytab().
• Wrong _HOST. Mismatched forward and reverse DNS yields "server not found in Kerberos database". Fix DNS rather than hard-coding principals.
• Open znodes after enabling security. Check ACLs under /hbase once the cluster is secure.
• Two authorizers. AccessController and Ranger configured together give confusing double decisions. Use one.
• Audit flood. Auditing every scan on hot tables can overwhelm Solr. Exclude trusted service users from audit where policy allows, and size the audit store.
• Superuser lockout. A bad policy or grant can lock out everyone except hbase.superuser entries. Keep that list small but never empty, and store its keytab offline.

Trade-offs

What to do next

1. Inventory every client and gateway that talks to HBase, and the identity each runs as today.
2. Confirm NTP, forward and reverse DNS, and a KDC reachable from every host before enabling Kerberos.
3. Secure ZooKeeper with SASL first, then HBase, then check znode ACLs.
4. Choose AccessController or Ranger, not both; with Ranger, model namespaces per team and keep sensitive columns in their own family.
5. Enable impersonation on Thrift and REST so audits show real users.
6. Decide on wire protection: privacy for a quick win, or native TLS on 2.6+ with the five-step rolling enable.
7. Test the failures on staging: rotate a keytab, skew a clock, stop Ranger Admin, and run a Spark job longer than the token lifetime.