Why architecture matters here
The architecture matters because a Hadoop cluster is a mesh, not a client-server pair. A single job involves a client, the ResourceManager, the NameNode, dozens of DataNodes, a NodeManager per host, the Hive metastore, ZooKeeper, and possibly Kafka and HBase. Every one of those hops needs mutual authentication. A design where each service holds a password for each peer is combinatorially impossible to operate — the credential count grows as the square of the services. Kerberos exists precisely to collapse that: every party trusts one KDC, and the number of trust relationships becomes linear.
It matters because passwords cannot cross the network here and there are too many hops for them to. Kerberos never transmits a password; the KDC and the principal each know a key derived from it, and the protocol proves possession without disclosure. Tickets are then time-bounded and service-scoped, so a captured ticket is useful only for one service, for a few hours. That property is what makes it safe to authenticate hundreds of times a minute across a cluster whose internal network you cannot fully trust.
The structure matters because Hadoop's workloads break the ordinary Kerberos model in a specific way, and the fix is architectural. A MapReduce or Spark job spawns tasks on hundreds of nodes, each of which must talk to the NameNode as the submitting user, potentially hours after submission and long after the user's TGT expires. Shipping the user's TGT or keytab to every node would be both a credential-distribution nightmare and a serious compromise. Delegation tokens are Hadoop's answer, and they are not an optimization — without them, distributed jobs on a secured cluster simply could not run.
It matters because of a mundane detail with outsized consequences: Kerberos identities and Unix identities are different namespaces. A principal is alice/admin@CORP.EXAMPLE.COM; an HDFS ACL applies to the user alice. Something must bridge them, and that something is a set of regular expression rules called auth_to_local. Those rules are security-critical and are frequently treated as boilerplate copied between clusters. A sloppy rule that maps a broad set of principals to a privileged local user is a full authentication bypass hiding inside what looks like a formatting config.
Finally, the architecture matters because the KDC becomes a hard dependency of the entire cluster, which is a real availability decision. When the KDC is down, no new authentication succeeds — jobs cannot start, services cannot restart, and existing tickets expire on a clock that does not care about your incident. A single KDC is therefore a single point of failure for the whole platform, and this is why KDC replication and clock synchronization are not peripheral concerns but core cluster infrastructure with the same criticality as the NameNode itself.
The architecture: every piece explained
Principals and realms are the naming system. A principal is the identity: user principals look like alice@CORP.EXAMPLE.COM, and service principals carry a host — nn/host1.corp.example.com@CORP.EXAMPLE.COM. The host component is not decoration: it means the service ticket is bound to a specific host, and the client constructs the principal name from the hostname it resolved. This makes DNS part of the authentication path, which is the origin of a large share of Kerberos failures. The realm, conventionally uppercase, is the administrative domain and the boundary at which cross-realm trust becomes necessary.
The KDC has two logical halves. The Authentication Service takes a request from a principal and returns a ticket-granting ticket — a credential that proves you authenticated once, encrypted so only you can use it. The Ticket-Granting Service takes that TGT plus a request for a specific service and returns a service ticket for that service alone. The two-step structure is the entire point of Kerberos: you prove your identity once per session and thereafter obtain per-service tickets without touching your password again, which is what makes a hundred authentications a minute practical.
Keytabs solve the unattended case. A human types a password and gets a TGT; the NameNode cannot. A keytab is a file holding the principal's long-term key, letting a service authenticate non-interactively at startup and re-authenticate forever. This makes the keytab equivalent to the password: anyone who reads the NameNode's keytab can impersonate the NameNode. Keytab file permissions are consequently among the highest-value security controls in the cluster, and a keytab in a source repository or a config-management payload readable by the whole estate is a total compromise of that service's identity.
Delegation tokens are Hadoop's own layer on top, and they are what makes the model actually work at cluster scale. At job submission, the client — while it still holds a valid TGT — asks the NameNode for a delegation token: a bearer credential saying 'the holder may act as alice'. That token is passed to the ResourceManager and distributed to every task. Tasks present the token rather than a Kerberos ticket, which means no Kerberos credential ever leaves the client. Tokens have a lifetime and a maximum lifetime, and the ResourceManager renews them on the job's behalf until the ceiling — which is why a job running longer than the maximum token lifetime fails at hour seven with an error about an expired token, having run fine all night.
auth_to_local and the SASL layer complete the picture. Hadoop's RPC uses SASL with the GSSAPI mechanism to carry the Kerberos exchange, and once the peer's principal is verified, the hadoop.security.auth_to_local rules translate it to a local username via ordered regex rules — stripping the realm, mapping alice/admin to alice, or rejecting what matches nothing. The resulting username is what HDFS ACLs and Ranger policies see, and group membership is resolved separately on the NameNode via its own group mapping. The clean way to hold the whole model: Kerberos establishes the principal, auth_to_local converts it to a username, group mapping expands it, and only then does authorization begin.
End-to-end flow
Follow a user reading a file. Alice runs kinit, and her client sends an AS-REQ to the KDC naming her principal. The KDC responds with a TGT encrypted under a key derived from her password; her client decrypts it, proving she knows the password without ever sending it. The TGT lands in a ticket cache on local disk with a lifetime — typically ten hours — and a renewable window, often seven days. From here on, her password is not involved.
Alice runs hdfs dfs -cat. The client resolves the NameNode's hostname and constructs the service principal nn/namenode-host@CORP.EXAMPLE.COM, then sends a TGS-REQ to the KDC presenting her TGT and naming that service. The KDC returns a service ticket for the NameNode alone. This is where DNS enters the trust path: if reverse lookup yields a different hostname than expected, the client asks for a ticket for a principal that does not exist and the KDC returns 'server not found in Kerberos database' — an error that reads like a KDC problem and is a DNS problem.
The client opens an RPC connection and the SASL/GSSAPI handshake begins. It presents the service ticket; the NameNode decrypts it using the key from its keytab, which it loaded at startup. Successful decryption proves two things at once: the ticket was issued by the KDC, and this service is the one it was issued for. Mutual authentication is now complete — Alice knows she is talking to the real NameNode, and the NameNode knows it is talking to Alice.
The NameNode extracts the principal and runs it through auth_to_local. The rules strip the realm and yield the username alice. The NameNode then resolves Alice's groups through its configured group mapping — often a shell call to the OS or an LDAP lookup — and only now, with a verified username and group list, does authorization run: the HDFS permission check or the Ranger policy evaluation. Kerberos has finished its job entirely; everything from here is a different subsystem.
Now the distributed case, where the delegation token earns its place. Alice submits a Spark job. While she still holds a valid TGT, the client requests an HDFS delegation token and a token for every other service the job will touch. Those tokens go to the ResourceManager, which stores them and distributes them to each container. A task on node 200, six hours later, presents the delegation token to a DataNode and is recognized as Alice — without a TGT, without a keytab, without any Kerberos credential having crossed the network. The ResourceManager renews the tokens periodically to keep long jobs alive, but it cannot renew past the maximum lifetime, which is the ceiling that ends very long jobs abruptly.
The synthesis worth carrying: there are three distinct credentials in play, and knowing which one failed is most of the diagnosis. The TGT proves the human authenticated and lives in a cache with an expiry. The service ticket is scoped to one service and is what the SASL handshake carries. The delegation token is a Hadoop-specific bearer credential that lets tasks act as the user without Kerberos. An error at kinit is a KDC or password problem; an error at connection time is a ticket, DNS, or keytab problem; an error hours into a job is almost always a token lifetime problem. That three-way split turns Kerberos's famously opaque errors into a tractable decision tree.