Why it matters
Security auditors, regulators, and internal security teams all require authentication for data platforms holding sensitive data. Kerberos is what makes Hadoop authentication auditable and provably secure. Without it, HDFS access control is honor-system: the CLI trusts whatever username you claim, and there is no way to know if a job was submitted by who it says.
Kerberos also enables downstream security features. Ranger authorization, HDFS encryption zones, and secure inter-service communication all require Kerberos as the foundation. Without it, none of them work.
The architecture
Every Kerberos-secured service in Hadoop has a principal, which is a string like nn/hadoop-nn.example.com@REALM. The principal identifies the service and the host it runs on. Users have their own principals, typically like alice@REALM. Each principal has a secret (either a password for humans or a keytab file for services).
The Key Distribution Center is a trusted third party that mediates authentication. On kinit, the KDC issues a Ticket Granting Ticket (TGT) proving the user's identity. When the user needs to talk to a service, they exchange the TGT for a Service Ticket (ST) specific to that service. The ST is what actually authenticates each RPC.
How it works end to end
Cluster setup requires creating a principal for every service instance (NameNode, DataNode, ResourceManager, NodeManager, and one per host), storing each in a keytab file on the corresponding host, and configuring Hadoop to use it. The DataNode principal often uses HTTP/hostname to enable web UI Kerberos. Every service reads its keytab on startup and uses it to obtain tickets when talking to other services.
Client access works through kinit or a keytab. A user runs kinit to get a TGT and then HDFS or YARN commands automatically use it. Applications use UserGroupInformation.loginUserFromKeytab() to authenticate service accounts against a keytab file, so batch jobs work without human interaction.
Ticket lifetime is finite. TGTs default to 24 hours renewable up to 7 days. STs are shorter. Long-running services need to renew tickets before they expire, which UserGroupInformation handles automatically for Java clients.