Why architecture matters here
The architecture matters because the single-NameNode ceiling is a real, hard limit that large HDFS deployments hit, and the ways of pushing past it each have serious drawbacks that RBF is designed to avoid. Simply growing the NameNode heap buys time but eventually runs into JVM garbage-collection pauses that get longer as the heap grows, hurting latency for every operation. Splitting workloads across entirely separate clusters fragments the data and forces applications to know which cluster holds what. RBF lets you keep one logical namespace while distributing its load and capacity across many NameNodes.
It matters because it makes the namespace layout an operational decision rather than a client-configuration one. Under the older client-side federation, the mapping from paths to NameNodes lived in every client's config file, so rebalancing data across subclusters or adding a subcluster meant a coordinated config rollout to every application and user. With RBF the mount table lives in the cluster's State Store and is consulted by the routers, so an administrator can change the layout centrally — mount a new subcluster, remount a path — without touching any client. Clients keep using the same global paths throughout.
It matters for load distribution. Different parts of a namespace have wildly different access patterns: a /logs tree hammered by ingestion, a /ml tree read by training jobs, a /warehouse tree queried by analytics. Putting each on its own subcluster means each has its own NameNode heap and its own RPC capacity, so a burst of metadata operations against one part of the namespace does not starve the others by saturating a single shared NameNode. RBF turns 'one NameNode for everything' into 'a NameNode per workload domain, presented as one filesystem'.
Finally it matters because it introduces a layer that must itself be highly available and low-overhead, and understanding that trade is essential to deploying it well. The router adds a network hop and a mount-table lookup to every operation, so it must be fast and it must not become a new single point of failure. In practice that means running several routers behind a single endpoint, keeping them stateless so any router can serve any request, and backing them with a replicated State Store. The benefit — horizontal namespace scale with a stable client view — is only worth it if the router tier is engineered to be invisible in the latency budget and resilient to individual router loss.
The architecture: every piece explained
A subcluster is a complete, independent HDFS cluster: its own NameNode (typically HA, with active and standby) and its own set of DataNodes storing blocks. Subclusters do not know they are part of a federation; each serves its own slice of the namespace exactly as a standalone HDFS would. Federation is a layer above them, not a change to them, which is why existing clusters can be brought under RBF without re-architecting their internals.
The router is the proxy that presents the unified view. It implements the same client RPC protocol a NameNode does, so clients and tools talk to it exactly as they would to a NameNode. When a request arrives, the router resolves the requested path against the mount table, determines the owning subcluster, and forwards the RPC to that subcluster's NameNode, then relays the response back. Critically, routers are stateless with respect to filesystem data — they hold no namespace themselves — so you run several of them behind one virtual endpoint and any of them can handle any request.
The State Store is the router tier's shared source of truth. It holds two key things: the mount table (the global-path-to-subcluster mapping) and membership/health information about the subclusters and routers. Because it is shared and replicated (commonly backed by ZooKeeper or another reliable store), every router sees the same mount table and the same view of which subclusters are up, so routing decisions are consistent no matter which router a client reaches.
The mount table is the heart of the design. It is a set of entries mapping a global path prefix to a destination subcluster (and a path within it) — for example /data → subclusterA:/data, /logs → subclusterB:/logs, /ml → subclusterC:/ml. Longest-prefix matching resolves a given path to its owner. Mount points can also be configured to spread a single logical path across multiple subclusters (for balancing), and to control behavior like read-only mounts or how quotas are applied at the federation level.
Router quorum and the single endpoint complete the picture. Clients are pointed at a logical address (a DNS name or a configured set of routers) that fronts the router tier, so the loss of one router just means requests go to another. Routers periodically report their health and the subclusters' health to the State Store, and they cache mount-table and membership data to keep the per-request lookup cheap. The result is a highly available proxy layer that turns many independent NameNodes into one addressable filesystem.
End-to-end flow
Follow a client reading a file at /ml/models/current.bin. The client is configured to talk to the router endpoint, so its open RPC lands on one of the routers. The router takes the path and resolves it against the mount table it has cached from the State Store: longest-prefix matching finds the /ml entry pointing at subcluster C, and rewrites the path to C's local path if needed.
The router forwards the open RPC to subcluster C's active NameNode, which handles it exactly as it would for a direct client: it returns the block locations for the file — the DataNodes in subcluster C holding the file's blocks. The router relays that response back to the client. Note what does not happen: the actual file data does not flow through the router. Just like normal HDFS, the client now talks directly to subcluster C's DataNodes to stream the blocks. The router is on the metadata path only, which is why its overhead stays small.
Now suppose an administrator needs to add capacity because the namespace is growing. They stand up a new subcluster D, then add a mount-table entry — say /archive → subclusterD:/archive — in the State Store. From that moment every router sees the new entry, and clients writing under /archive are transparently routed to D's NameNode. No client configuration changed; the global namespace simply grew by one subcluster, and D's NameNode heap and RPC capacity are now available to the federation.
Consider the harder case: a client tries to rename /logs/2026/raw to /ml/2026/raw. The router resolves the source to subcluster B and the destination to subcluster C — the rename crosses a subcluster boundary. HDFS rename is an atomic metadata operation within a single NameNode; there is no atomic cross-NameNode rename. The router cannot make two independent NameNodes rename atomically, so it rejects the operation (or it must be handled by an explicit copy-then-delete that is not atomic). This is the sharp edge of federation: as long as related data lives within one subcluster, operations are fast and atomic, but operations that span the mount-table boundaries expose the fact that underneath the single namespace there are still several independent filesystems. Good mount-table design keeps things that move together in the same subcluster precisely to avoid this.
It is worth tracing what the router does under load to see why the extra hop stays cheap. Because routers cache the mount table and subcluster membership from the State Store, the per-request work is just a longest-prefix lookup in an in-memory structure plus the forwarding of a single RPC — no disk access, no coordination. A router handling a metadata-heavy client adds a small, roughly constant latency to each namespace operation, and since the bulk data path bypasses it entirely, even a client streaming terabytes of file content puts almost no load on the router tier. This asymmetry is what lets a modest number of routers front a very large federation: their capacity is sized against the aggregate metadata operation rate, which is far smaller than the aggregate data throughput. When routers do become a bottleneck it is almost always because the metadata rate has grown, and the fix is simply to add more stateless routers behind the same endpoint — a horizontal scale-out that mirrors the way subclusters scale the namespace itself. The two scaling axes are independent: routers absorb metadata request growth, subclusters absorb namespace size growth, and an operator can dial each one without disturbing the other.