Why architecture matters here
A distributed filesystem shared by many tenants has no natural back-pressure against a single greedy consumer. Disk is a global, exhaustible resource: once the DataNodes fill up, every write across the cluster fails, not just the writes of the team that caused it. Without quotas, capacity is governed by social convention and after-the-fact cleanup, which means the cluster's stability depends on no one ever making a mistake — an assumption that fails the first time a misconfigured job writes a billion tiny files or an unbounded log directory grows without rotation. Quotas convert a shared, first-come-first-served resource into a set of enforced, per-tenant budgets.
The two failure modes quotas address are distinct, which is why HDFS separates namespace from space. Namespace pressure is about object count: the NameNode holds the entire filesystem tree in memory, and every file and directory is an INode consuming heap. A job that creates hundreds of millions of tiny files can exhaust NameNode heap and destabilize the whole cluster long before it fills any disk — so a namespace quota caps the count of INodes in a subtree regardless of their size. Space pressure is about bytes on disk, and a space quota caps consumed storage. A directory can hit either limit independently: a subtree of a million empty files trips the namespace quota; a single enormous file trips the space quota.
Storage-type quotas add a third dimension that matters once a cluster tiers its storage. When hot data lives on SSD and cold data on ARCHIVE nodes, the SSD tier is a scarce, expensive resource that a single tenant could monopolize by marking everything hot. A storage-type quota caps how many bytes a subtree may place on each specific medium, so the fast tier is rationed deliberately rather than claimed by whoever writes first. Architecturally, the reason all of this belongs in the NameNode — rather than a monitoring dashboard that emails you when you are over — is that only the NameNode sits on the single serialization point where every namespace change is committed, so it is the one place enforcement can be both authoritative and synchronous.
A further reason quotas belong to the filesystem rather than a scheduler or a job framework is that storage is consumed outside any single job's lifetime. A YARN queue can bound the CPU and memory a job uses while it runs, but the files that job leaves behind persist indefinitely and keep consuming disk long after the job exits. Only a filesystem-level budget can govern that durable footprint, and only the NameNode knows the true, up-to-the-moment consumption of every subtree. This is why quotas and compute-side resource controls are complementary rather than redundant: the scheduler rations the rate of work, and quotas ration the durable state that work produces, and a healthy multi-tenant cluster needs both bounded at once.
The architecture: every piece explained
Quotas are attributes attached to directory INodes in the NameNode's in-memory tree; files do not carry quotas, only the directories that contain them. A directory with a quota set becomes a quota accounting point, and the NameNode maintains a QuotaCounts structure for it holding the current consumption: the number of namespace objects (files + directories) in the subtree, the total space consumed, and a per-storage-type byte count. These counts are the crux of the design — they are maintained incrementally so that a check is an O(1) comparison rather than an O(subtree) walk.
The space quota counts replicated bytes, not logical bytes: a 1GB file with replication factor 3 consumes 3GB against the space quota, because that is the real disk it occupies across the cluster. This is a frequent source of surprise — raising a file's replication factor consumes more quota even though the logical data is unchanged, and a setReplication that would push a directory over its space quota is rejected. Erasure-coded data is counted by its actual on-disk footprint (data plus parity cells) rather than a naive replication multiple. The namespace quota counts objects: every file and every directory in the subtree is one unit, so the minimum namespace quota for a directory is 1 (the directory itself).
The storage-type quota works together with storage policies. A storage policy (HOT, WARM, COLD, ALL_SSD, etc.) determines which storage types a file's replicas target; the storage-type quota then caps the bytes allowed on each type within a subtree. The two are complementary: the policy expresses intent about placement, and the quota enforces a ceiling on that placement. When a block is written, the NameNode attributes its bytes to the storage type its replicas land on and charges them against the relevant type quota.
An important property of the design is that all three quota types are checked independently and conjunctively: a mutation must satisfy every applicable quota on every ancestor, and hitting any one of them rejects the operation. This means a directory can be nowhere near its space quota yet blocked because its namespace quota is exhausted, or have plenty of namespace headroom yet fail because the SSD storage-type quota is full. The independence is deliberate — the three limits guard three genuinely different scarce resources (NameNode heap, total disk, and fast-tier disk), and collapsing them into one number would lose the ability to reason about each. It also means diagnosing a rejection requires reading all three consumptions, not assuming the obvious one; the QuotaExceededException subtype (namespace versus disk) tells you which wall was hit, and dfs -count -q shows the full picture so an operator can raise the specific limit that is binding rather than guessing.
Enforcement lives in the namespace mutation path. When a client issues a mutating RPC, the NameNode resolves the target path to its INode, then walks up to find every ancestor directory that has a quota and verifies the pending change against each QuotaCounts. If any ancestor would be exceeded, the operation aborts with a NSQuotaExceededException or DSQuotaExceededException and nothing is committed. On success, the counts are updated incrementally up the ancestor chain. Administration flows through hdfs dfsadmin -setQuota / -setSpaceQuota / -setStorageTypeQuota and their clear counterparts, while hdfs fsck, dfs -count -q, and dfs -du report configured limits against actual consumption.
End-to-end flow
Trace a write into a quota-bounded directory. A Spark job calls create on /data/team-a/output/part-00042. The NameNode resolves the path and walks the ancestor chain: /data/team-a has a namespace quota of 10 million and a space quota of 50TB set on it. Before allocating the new INode, the NameNode checks the directory's cached QuotaCounts: current namespace usage is 9,999,998, so adding one file (and, if needed, one directory) stays under 10 million — the namespace check passes. The file is created empty, and the namespace count is incremented up the chain.
Now the job writes data. As blocks are allocated, the NameNode charges replicated bytes against the space quota: each 128MB block at replication 3 is 384MB of consumed space. Suppose /data/team-a is at 49.6TB consumed. The job keeps writing; when the next block allocation would push consumption past 50TB, the NameNode rejects the block with a DSQuotaExceededException. The client's write fails cleanly — the job errors out at its own budget boundary — and, critically, no other tenant is affected because the DataNodes still have free capacity that team-a simply is not allowed to claim.
Consider a storage-type interaction. The output directory carries an ALL_SSD storage policy and /data/team-a has an SSD storage-type quota of 5TB. As replicas target SSD volumes, their bytes are charged against the SSD type quota. Once the subtree has placed 5TB on SSD, further SSD placement is rejected even though the overall space quota still has room — so new writes either fail or fall back to another tier depending on policy. This is the mechanism that keeps the expensive fast tier rationed across tenants rather than consumed by whoever writes hottest.
Finally, an administrative correction. An operator notices team-a is chronically near its space quota and raises it with hdfs dfsadmin -setSpaceQuota 80t /data/team-a. The change updates the directory's quota attribute immediately; the next write that previously would have been rejected now succeeds against the higher ceiling. To audit before deciding, the operator ran hdfs dfs -count -q -h /data/team-a, which printed the namespace and space quotas alongside remaining headroom, and hdfs dfs -du -h to see where the bytes actually went — the reporting path and the enforcement path read the same QuotaCounts, so the numbers reconcile.
The interaction between quotas and HDFS snapshots is a frequent operational trap worth tracing explicitly. Snapshots are cheap, copy-on-write views of a directory at a point in time; deleting a file from a snapshotted directory removes it from the live tree but the snapshot still references its blocks, so those blocks are not reclaimed and their bytes still count against the directory's space quota. From the operator's perspective this looks paradoxical: a team deletes terabytes, sees no free space return, and reports a 'quota bug.' The mechanism is working as designed — the space is legitimately still consumed because the snapshot preserves it. The remedy is to account for snapshot retention when sizing quotas and to delete or age out snapshots as part of capacity management, so that space quota reflects the true retained footprint rather than only the live tree. The same reasoning applies to the trash directory, whose retained deletions consume quota until purged.