Azure Data Lake Storage Gen2, in depth: the hierarchical namespace, dfs and blob endpoints, ABFS, and the RBAC plus ACL security model

What the hierarchical namespace actually changes

A flat object store has no directories. The key raw/sales/2026/09/part-0001.parquet is a string; the slashes are a naming convention that list operations simulate with prefix and delimiter queries. That works for reads but breaks down for the operations analytics engines rely on. Renaming a directory means copying every object under the prefix and deleting the originals, one request per object and not atomic: a crash halfway leaves some files in each place. Deleting a directory is the same loop. Permissions can only be expressed per container or per object, not for a subtree. The object storage deep dive covers why object stores are built this way.

With HNS enabled, the account keeps a real namespace tree. A directory is an entry with an owner, an owning group, permissions and ACLs, and files are children of it. Renaming or deleting a directory becomes a metadata operation on the tree: it is atomic and its cost does not grow with the number of files underneath. This is the property Hadoop-style output committers were designed around. A job writes into a temporary directory and commits by renaming it, and on a flat store that rename becomes a slow, non-atomic copy, which is why S3 needed special committers.

Enabling HNS is an account-wide decision, made at creation or by upgrading an existing account. The upgrade is one-way, and writes are disabled while it runs, so validate it on a non-production copy first. Some Blob Storage features have behaved differently on HNS accounts over time; check the current feature-support table rather than assuming parity.

Two endpoints, one account, and the ABFS driver

An HNS account exposes two REST surfaces over the same data. The blob endpoint (<account>.blob.core.windows.net) speaks the Blob API: containers and blobs. The dfs endpoint (<account>.dfs.core.windows.net) speaks the Data Lake Storage REST API: filesystems, directories and files, with operations such as create path, rename, set ACL and get properties. A container in the blob API is a filesystem in the dfs API. Because both views share one namespace, a file written through one endpoint is readable through the other; this is what Microsoft calls multi-protocol access.

Hadoop, Spark and most engines built on them use the ABFS driver, part of Apache Hadoop's hadoop-azure module. It is a thin shim over the dfs REST API, which is exactly why it replaced the older WASB driver: WASB had to fake directory semantics on top of blobs, with the same copy-and-delete rename problem. ABFS addresses data with URIs of the form abfss://<filesystem>@<account>.dfs.core.windows.net/<path>, where abfss uses TLS and should be the only scheme you allow. The driver authenticates with either Shared Key or Microsoft Entra ID OAuth tokens (a service principal, a managed identity or a user); with OAuth, every call is authorized against the ACLs for the calling identity.

<!-- core-site.xml fragment: ABFS with a managed identity instead of an account key -->
<property>
  <name>fs.azure.account.auth.type.mylake.dfs.core.windows.net</name>
  <value>OAuth</value>
</property>
<property>
  <name>fs.azure.account.oauth.provider.type.mylake.dfs.core.windows.net</name>
  <value>org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider</value>
</property>

Check property names against the hadoop-azure documentation for your Hadoop version. The point: configure identities, not keys, so ACLs apply.

The file write model: append, then flush

The dfs API writes files in two steps. You create the path, append bytes at explicit offsets (possibly in several calls, possibly in parallel from one writer), and then flush at a final position, which commits everything up to that position and makes it visible. Until the flush, appended data is not part of the file. The Python SDK exposes this directly:

from azure.identity import DefaultAzureCredential
from azure.storage.filedatalake import DataLakeServiceClient

svc = DataLakeServiceClient("https://mylake.dfs.core.windows.net",
                            credential=DefaultAzureCredential())
fs = svc.get_file_system_client("curated")

# Write a batch into a staging directory, then publish it with one atomic rename.
staging = fs.get_directory_client("sales/_staging/batch-20260930")
staging.create_directory()
f = staging.create_file("part-0001.parquet")
data = open("part-0001.parquet", "rb").read()
f.append_data(data, offset=0, length=len(data))
f.flush_data(len(data))                       # commit: now visible to readers

staging.rename_directory(new_name="curated/sales/dt=2026-09-30")

The pattern in the last line is the reason to use HNS. Readers never see a half-written partition: the directory appears under its final name all at once, whatever number of files it contains. Note that rename_directory takes the new name including the filesystem prefix. Renames within an account are metadata operations; there is no rename across accounts, which is a copy.

How authorization is evaluated

ADLS supports five authorization mechanisms: Shared Key, shared access signatures (SAS), Azure RBAC, Azure ABAC conditions on role assignments, and POSIX-style ACLs. The first two are special. Shared Key and account or service SAS carry no identity, so RBAC and ACLs do not apply at all; Shared Key is effectively super-user access, including changing owners and ACLs. A user delegation SAS is the exception: it is backed by Entra credentials, and ACLs can be checked against it.

For identity-based requests the order is fixed. Azure first checks for a role assignment such as Storage Blob Data Reader, Contributor or Owner on the account or container. If one exists and has no ABAC conditions, access is granted. If it has conditions and all of them match the request, access is granted. Only when no role grants access are the ACLs evaluated. The consequence is the most misunderstood rule of ADLS security: an ACL cannot restrict access that a role assignment already grants. Give a pipeline Storage Blob Data Contributor on the account and every carefully set ACL is irrelevant to it.

ACL evaluation mirrors POSIX and HDFS (see HDFS permissions and ACLs for the lineage). Identities are checked in order: super-user, owning user, named users and service principals, then owning group and named groups, then everyone else. The first matching identity decides, with named groups considered together. The mask caps the effective permissions of named users, named groups and the owning group, not of the owning user. On directories, read plus execute lists contents, write plus execute creates children, and execute alone lets you traverse. That last point is the source of most 403s: to read one file with ACLs alone, the caller needs execute on the container root and on every directory on the way down, plus read on the file.

Access ACLs, default ACLs and why existing files do not change

Every file and directory has an access ACL, which governs access to that item. Directories also have a default ACL, a template copied into the access ACL (and, for subdirectories, the default ACL) of each child created afterwards. Files never have default ACLs. Changing a default ACL affects only future children; nothing that already exists changes. Permissions are stored on each item, not inherited at evaluation time. When no default ACL is set, new items get permissions derived from the parent and a fixed umask of 007, so the other class gets nothing.

To change permissions on an existing tree you must apply the change recursively, and the SDKs provide operations for that. They work in batches, return continuation tokens, and report a count of failures rather than stopping at the first, so treat them as a job you check, not a call you fire and forget:

from azure.storage.filedatalake import AccessControlChangeResult

root = fs.get_directory_client("sales")
acl = "group:4f1d...-analysts:r-x,default:group:4f1d...-analysts:r-x"

result: AccessControlChangeResult = root.update_access_control_recursive(acl=acl)
c = result.counters
print(c.directories_successful, c.files_successful, c.failure_count)
if c.failure_count:
    # Retry from result.continuation, or inspect per-path failures via a progress hook.
    raise RuntimeError(f"{c.failure_count} paths were not updated")

Two limits shape the design. Each access ACL and each default ACL holds at most 32 entries, effectively 28 once the owning user, owning group, mask and other entries are counted. And Entra recommends that a principal belong to fewer than 200 groups, because group membership travels in the token and larger sets can cause performance problems. Both point the same way: put groups in ACLs, never individual users or service principals, and grant access by changing group membership. Use the object ID of the service principal, not of the app registration, when you do add one; they are different GUIDs and the wrong one silently grants nothing.

Worked example: a three-zone lake with least privilege

Consider an account mylake with HNS, Shared Key access disabled, and three filesystems: raw, curated and sandbox. An ingestion pipeline lands data in raw, a Spark job transforms raw into curated, analysts read curated, and data scientists get a private area in sandbox. No principal gets a data-plane role on the account; all data access is by ACL.

Execute on each filesystem root is the minimum that lets a group reach its folder without listing its siblings. Default entries on each working folder mean files written tomorrow get the same ACL as files written today. The transform job's managed identity is added to transform-jobs, not named in the ACL, so rotating to a new identity is a group change. If analysts later need a new subfolder, you add a default entry at the right level and run a recursive update for existing data, then check the failure count.

The common way this design breaks: someone gives the transform job Storage Blob Data Contributor to fix a 403 quickly, and from then on it can write anywhere in the account, whatever the ACLs say.

Failure modes and how to diagnose them

• 403 on a file you granted read to. A parent directory lacks execute for that identity. Walk up the path and check each level.
• New files readable, old files not. A default ACL was added after the data was written; run a recursive update.
• ACLs seem ignored. Either the caller has a role assignment that already grants access, or it authenticates with Shared Key or a service SAS. Disable Shared Key authorization on the account if nothing needs it.
• Service principal gets nothing. The ACL names the app registration's object ID instead of the service principal's.
• Private endpoint works for some tools only. The blob and dfs endpoints are separate hostnames; a network design that resolves only one privately breaks ABFS or blob tools. Provision both.
• Slow queries. Millions of small files each cost requests; compact them, as in the small files guide.
• Partial recursive updates. A recursive ACL job hit failures or timed out; resume from the continuation token and alert on non-zero failure counts.

Operating it well

Turn on resource logs for the storage account and ship them to a workspace: they record the operation, the authentication type and the caller identity, which is how you find Shared Key use and 403 patterns. Enable soft delete, since an atomic directory delete is equally atomic when it is a mistake. Lifecycle rules can move cold partitions to cooler tiers, but archived files need rehydration before any engine can read them. Put table formats such as Delta Lake on top for transactions across files; HNS gives you atomic directory operations, not multi-file transactions or schema enforcement.

Trade-offs

HNS is the right default for analytics accounts: atomic renames, real directories and ACLs outweigh the cost of a namespace. For web assets or backups served only through blob APIs, a flat namespace is simpler. The RBAC plus ACL model has one sharp edge, roles overriding ACLs, and one operational cost: ACL changes to existing data are jobs. Groups everywhere and no data-plane roles on the account avoid most of the pain.

What to do next

1. Confirm HNS is enabled on every analytics account; plan the one-way upgrade in a test copy for any that are not.
2. Switch ABFS and SDK clients to Entra identities (managed identity or service principal) and disable Shared Key authorization.
3. List every data-plane role assignment on lake accounts and remove those that bypass your ACL design.
4. Create reader and writer groups per zone and grant access only through group ACL entries, with default entries on working folders.
5. Check that each group has execute on every parent path it needs, and run recursive updates for existing data with failure alerts.
6. Adopt a stage-then-rename commit for batch outputs, or a table format that handles commits for you.
7. Provision private endpoints for both blob and dfs, enable resource logs and soft delete, and review lifecycle rules before archiving anything engines read.