WebHDFS and HttpFS — REST APIs for HDFS Access from Anywhere

Why HDFS needs an HTTP surface at all

The native access path is genuinely good and genuinely narrow. A client opens an RPC connection to the NameNode (port 8020 by convention), gets block locations back, then opens data-transfer connections to individual DataNodes. It is efficient, it is topology-aware, and it requires a JVM. libhdfs is a JNI shim that still starts a JVM underneath, so it does not escape the constraint so much as hide it.

That leaves a long list of consumers stranded. Data science teams want a requests call, not a native-library build chain. Web dashboards want to serve a small HDFS file to a browser without a bespoke middle tier. Partner integrations and cross-datacenter transfers want something that survives a corporate proxy and a change-controlled firewall, which HTTP does and a family of high-numbered RPC ports does not. Ingestion tooling written in Go or Node has no other option at all.

There is a governance argument too, and it is usually the one that decides the deployment shape. HTTP is trivially loggable. A gateway that terminates every read and write produces one access log with one identity per line, which is the difference between an afternoon and a quarter when an auditor asks who read a file. That funnel is the real product HttpFS sells, and it is why organisations with a compliance obligation reach for the slower of the two designs on purpose.

The URL contract — one path prefix, one op parameter

The whole API is one URL shape. Every request is http://<host>:<port>/webhdfs/v1/<hdfs-path>?op=<OPERATION>, with the absolute HDFS path appended directly after the prefix and everything else carried as query parameters. There is no path-per-resource REST modelling here; the operation is a parameter and the HTTP verb is chosen to match its semantics.

The verb mapping is consistent enough to memorise. GET covers reads and metadata: OPEN, LISTSTATUS, GETFILESTATUS, GETCONTENTSUMMARY, GETFILECHECKSUM, GETACLSTATUS, GETXATTRS, CHECKACCESS. PUT covers creation and metadata mutation: CREATE, MKDIRS, RENAME, SETOWNER, SETPERMISSION, SETREPLICATION, SETTIMES, SETACL, CREATESNAPSHOT. POST is the small set that adds to existing data -- APPEND, CONCAT, TRUNCATE. DELETE is DELETE and DELETESNAPSHOT. Arguments ride alongside: overwrite, blocksize, replication, permission, recursive, destination, and for reads offset, length and buffersize.

Responses are JSON, and so are errors. A failure returns a RemoteException object carrying exception, javaClassName and message, mapped onto a status code -- a missing path is 404, an AccessControlException is 403, a malformed argument is 400. Teach your client to parse that body rather than branching on the status alone, because several very different problems arrive as the same 403.

Default ports are worth committing to memory because half the internet still quotes the Hadoop 2 numbers: NameNode HTTP is 9870 and DataNode HTTP is 9864 in Hadoop 3, where they were 50070 and 50075 before; the TLS pair is 9871 and 9865; HttpFS listens on 14000. On the Java side the same protocol is reachable as a filesystem, so hdfs dfs -ls webhdfs://nn-host:9870/data works from any client that can talk HTTP, with swebhdfs:// for the TLS variant.

The two-step redirect and the reason it exists

This is the mechanism everything else follows from. A read starts as a GET with op=OPEN against the NameNode, and the NameNode does not return the file. It returns 307 Temporary Redirect with a Location header naming a DataNode, carrying the operation forward along with a namenoderpcaddress parameter and the offset. The client follows the redirect and gets 200 with the bytes. Two round trips: one for metadata, one for data.

The reason is the same principle that shapes native HDFS. If the NameNode served file content, every byte the cluster reads would cross the one process that also holds the entire namespace in heap and answers every metadata call. HDFS separates the metadata path from the data path precisely so the NameNode never becomes the bottleneck, and WebHDFS preserves that separation over HTTP rather than quietly abandoning it. The NameNode does over REST exactly what it does over RPC: it resolves where the blocks are and hands back an address.

Which DataNode you get is not arbitrary. For a read the NameNode selects a node holding a replica of the first block, biased by network topology relative to the requesting address; for CREATE it chooses a target for the first block the way it would for any writer. A client whose address is not in the cluster's topology map -- which is every client outside the cluster -- gets an effectively arbitrary node, which turns out to matter later for balance.

One detail surprises people: the redirect DataNode is not merely handing over blocks it happens to store. It opens an HDFS client of its own and streams the whole file, fetching any block it does not hold from whichever DataNode does. A multi-block read through WebHDFS makes the redirect target a proxy for the remainder of the file.

On writes the two steps are explicit, and the tooling shows it. curl -i -X PUT "http://nn:9870/webhdfs/v1/tmp/f?op=CREATE" returns the 307; a second curl -i -X PUT -T localfile "<Location>" returns 201 Created. The split is deliberate -- it stops a client sending a body to the NameNode before it learns where the body belongs. Where a client handles redirects badly, noredirect=true returns the target address in a JSON Location field with a normal 200 instead, so the caller can issue the second request itself.

What the redirect costs at the network edge

Start with the symptom, because it is the most-reported WebHDFS problem and it never looks like a redirect problem. A curl that works perfectly from an edge node hangs or fails with a DNS error from a laptop, a VPN, or another datacenter. The cause is the Location header: it names the DataNode by whatever address that node advertises, which is routinely an internal hostname or a private IP the remote client cannot resolve and could not route to if it could.

The requirement this creates is worth stating carefully, because it is commonly written down backwards. WebHDFS traffic is client-initiated end to end; no DataNode ever dials back to the client, and nothing needs an inbound path to the client at all. What is required is that the client can reach every DataNode's HTTP port, and resolve whatever name the redirect contains. That is one firewall rule and one DNS entry per DataNode, and it grows every time the cluster does. On a 400-node cluster that is a network change request nobody wants to own.

dfs.datanode.use.datanode.hostname is the lever that decides whether hostnames rather than raw IPs are used for DataNode addressing, which makes split-horizon DNS a workable answer -- and is useless if the name resolves inside the cluster and nowhere else. Test the resolution from the network the clients actually live on before declaring the endpoint working.

A load balancer does not rescue this. Put one in front of the NameNodes and it will happily terminate the metadata call, but the Location header it passes back still names a DataNode directly, and the client leaves the balanced address on the next hop. Unless something actively rewrites that header, the client goes direct. That is exactly the gap HttpFS and Knox fill.

The protocol is also stateless: there is no session and no file handle. A connection that drops forty gigabytes into a large file is resumed only by reissuing OPEN with the right offset, and the bookkeeping is the client's problem.

HttpFS — one gateway, one hop, one bottleneck

HttpFS is a standalone service that implements the same REST API on port 14000. Same /webhdfs/v1 prefix, same operations, same filesystem scheme, so webhdfs://httpfs-host:14000/ works with clients written against native WebHDFS and no code changes. That compatibility is the point: the deployment decision stays an operator decision rather than leaking into every application.

Underneath it is completely different. HttpFS authenticates the caller and then acts as an HDFS client itself -- RPC to the NameNode, data transfer to the DataNodes -- on the caller's behalf. Nothing is redirected. The client's TCP connection terminates at the gateway and every byte returns through it. One host, one port, one certificate, one access log, one firewall rule.

One correction to older documentation and to any runbook inherited from it: HttpFS was long shipped as a Tomcat web application, and descriptions of it as a Tomcat service date from that era. Current builds run it on an embedded servlet container, so do not plan a deployment around a Tomcat installation you have to manage separately.

The trade-off is unambiguous. Everything the platform reads and writes through the gateway crosses a single JVM's heap and a single NIC. A 10 GbE host caps out around a gigabyte a second in the best case, and the servlet thread pool and heap bound concurrent streams well before the network does. Native WebHDFS' redirect exists specifically to avoid that ceiling, so choosing HttpFS is choosing to accept it in exchange for the operational simplification.

Requests are independent, so two or more instances behind a load balancer scale reads horizontally and remove the single point of failure. The one thing to get right is the authentication cookie: if signed cookies are issued after authentication, the signing secret must be shared across instances, or a client bounced to a different backend re-authenticates on every request. It is also worth knowing that HttpFS tracks the WebHDFS operation set rather than defining it, and newer operations have historically landed there later, so a modern client can get a 400 from an older gateway for an operation that works fine against the NameNode.

Knox and the third shape — rewriting the Location header

There is a third deployment that is neither of the two, and it is what most large regulated clusters actually run. A perimeter gateway speaks WebHDFS to the client, forwards the call into the cluster, and rewrites the redirect. Apache Knox is the usual implementation.

The mechanism is the interesting part. The client calls Knox at a single external hostname under a gateway path. Knox forwards the metadata request to the NameNode, receives the 307, and rewrites the Location header so it points back at Knox with an encoded reference to the chosen DataNode. The client follows the redirect -- to Knox again -- and Knox proxies the data leg. From outside, one hostname, one port, one certificate; inside, the ordinary two-step WebHDFS exchange, untouched.

That means Knox pays the same proxy cost on the data path that HttpFS does, so it buys nothing in throughput. What it buys is perimeter behaviour HttpFS does not offer: one front door for many Hadoop services rather than one per service, integration with an enterprise identity provider so external clients need no Kerberos configuration at all, and a single place to apply rate limits and request-level policy. The price is another service to run and another place where a URL rewrite rule can be subtly wrong -- a rewrite that misses a header produces a client that follows a redirect straight into an unreachable internal address, which looks identical to the plain firewall problem.

Worth noting if you already run Router-Based Federation: a router exposes WebHDFS on its own HTTP port and redirects to DataNodes across subclusters, so you get a REST endpoint that hides the namespace split without adding another tier.

Choosing between them

The decision follows almost entirely from where the clients sit and how much data moves.

Clients inside the cluster network, throughput matters. Native WebHDFS. The redirect is a feature here: reads spread across DataNodes, there is no shared bottleneck, and the connectivity requirement is already satisfied.

A handful of clients outside, moderate volume. HttpFS. Opening one port to one host is a change request that gets approved; opening every DataNode's HTTP port to an external network is one that should not be.

Many external clients, SSO required, several Hadoop services exposed. A perimeter gateway, quite reasonably in front of HttpFS rather than instead of it.

Interactive tools that list and read small files -- notebooks, file browsers, admin UIs. HttpFS, comfortably. Metadata calls dominate the workload and the proxy hop is invisible next to the round-trip latency.

Bulk cross-cluster copying. Neither, as a hand-rolled REST loop. Use DistCp, which will happily take a webhdfs:// source when the two clusters cannot agree on an RPC version -- that combination is a genuinely standard migration technique, and it gets you the listing, retry and verification machinery you would otherwise reimplement badly.

The anti-pattern is worth naming: WebHDFS as the file-serving API behind a public application. Millions of small object requests against the NameNode is the small files problem arriving through a new door, with per-request metadata cost and no caching layer. Put an object store or a CDN in front and let HDFS be the batch substrate it is.

Authentication — why user.name is not security

Three modes exist and only two are credentials.

Pseudo, or simple, authentication takes the identity from a query parameter: ?user.name=alice. Nothing verifies it. Anyone who can reach the port is whoever they say they are, including the HDFS superuser, and the permission model underneath is enforcing rules against a self-declared identity. It is a convenience for a development cluster and it is not an access control. The correct mental test: enable it only where you would be equally comfortable letting the same network run arbitrary hdfs dfs commands unauthenticated.

Omit user.name under simple auth and the request runs as the configured static user, hadoop.http.staticuser.user, whose default is dr.who. That is why Permission denied: user=dr.who is almost everyone's first WebHDFS error. It is not a bug and it is not a permissions problem -- it means no identity was sent at all.

Kerberos is the real one, carried over HTTP by SPNEGO. The server answers with WWW-Authenticate: Negotiate, the client replies with a service ticket for the HTTP/<host> principal, and the exchange completes before the operation runs. dfs.web.authentication.kerberos.principal and its matching keytab setting configure the HTTP identity the NameNode and DataNodes present; HttpFS has its own httpfs.authentication.type for the same purpose. From the command line it is curl --negotiate -u : with a valid ticket in the cache. The wider setup -- KDC, principals, keytab distribution -- belongs to Hadoop Kerberos.

After a successful handshake the server issues a signed cookie so following requests skip the negotiation, which matters because SPNEGO is not cheap and a file browser issues a lot of requests. The signing secret has to be consistent across every host expected to honour that cookie.

Authentication is not authorization, and the good news is that there is nothing new to learn. WebHDFS and HttpFS enforce exactly the permissions and ACLs the RPC path enforces, and if the cluster runs Ranger, its policies apply to REST access identically. There is no separate REST permission model to keep in sync.

Delegation tokens, impersonation and proxy users

SPNEGO needs a live Kerberos ticket, and a container that runs for eight hours does not reliably have one for its whole life. Delegation tokens are the answer: GET ?op=GETDELEGATIONTOKEN returns a token, subsequent calls pass it as ?delegation=<token>, and RENEWDELEGATIONTOKEN and CANCELDELEGATIONTOKEN (both PUT) manage its lifetime. A token is a bearer credential with a defined expiry and a nominated renewer, which is exactly the shape a long-running job needs.

There is a sharp edge here specific to the redirect. In a secure cluster the NameNode has to give the DataNode a way to authenticate the second leg without a fresh SPNEGO round trip, and it does so by embedding a delegation token in the redirect URL. That token is now part of a URL, and URLs end up in DataNode access logs, reverse-proxy logs, shell history and monitoring pipelines. Treat those logs as credential-bearing material with the retention and access rules that implies, and run TLS so the token is not also crossing the network in the clear.

Tokens expire, and the failure is unhelpfully shaped. A transfer that runs longer than the token lifetime dies partway through with an authentication error, which reads like a configuration problem rather than the timeout it is. Plan renewal, or size the work to finish inside the window.

Impersonation is the other half. A gateway -- HttpFS, Knox, or anything else fronting HDFS -- authenticates the end user but connects to HDFS under its own service identity. For that to work the NameNode must be told the service identity may act on behalf of others, via hadoop.proxyuser.<service>.hosts and hadoop.proxyuser.<service>.groups in core-site.xml. Both default to empty, so a freshly installed gateway fails with an impersonation error until they are set -- and setting either to * grants that host the ability to be anyone, which makes the gateway host's own security posture the cluster's security posture. Restrict by host and by group, and mean it. On the request side, doas is how a caller explicitly asks to act as another user, subject to the same proxy rules.

TLS, swebhdfs and the hardening knobs

dfs.http.policy selects between HTTP_ONLY, HTTPS_ONLY and HTTP_AND_HTTPS, and the swebhdfs:// scheme requires TLS to be enabled. The TLS ports are 9871 on the NameNode and 9865 on the DataNode.

Certificates are where native WebHDFS over TLS most often breaks, and the redirect is again the culprit. Every DataNode needs a certificate valid for the exact name that appears in the Location header. Issue the certificate for the internal FQDN, have the redirect emit an IP address, and hostname verification fails on the second hop with an error that says nothing about redirects. Multiply that by the DataNode count and you have a certificate lifecycle problem that recurs on every rotation. HttpFS collapses this to one certificate on one host, which is a real and frequently decisive operational argument for the gateway.

Cross-site request forgery is a genuine concern the moment a browser holds an authenticated session against the endpoint, because a state-changing operation is a single URL. dfs.webhdfs.rest-csrf.enabled requires a custom header -- X-XSRF-HEADER by default -- on requests that change state, which a hostile page cannot add cross-origin. HttpFS has the equivalent setting. It costs one header in your client and it closes the hole; turn it on if browsers touch the endpoint at all.

One correction to a common framing: dfs.webhdfs.enabled defaults to true. WebHDFS is not something you switch on so much as something already listening, and the interesting configuration decision is whether to turn it off. If the sanctioned path is a gateway, and WebHDFS is still answering on every NameNode and DataNode, then any client with network reach has bypassed the audit funnel the gateway exists to provide. Disabling it on the cluster nodes is a legitimate hardening step, not paranoia.

Encryption zones deserve a moment of thought as well. Native clients decrypt on the client side after fetching the key; over REST the decryption happens server-side, on the redirect DataNode or the gateway, and plaintext then crosses HTTP. TLS covers the wire, but the trust boundary has moved, which is worth reconciling against whatever threat model justified the encryption zone in the first place.

Throughput, replica placement, and the balance you quietly destroy

REST access is slower than the native path, and it is worth knowing precisely why rather than treating it as folklore.

First, no short-circuit reads. A native client co-located with a DataNode can be handed a file descriptor and read the block file directly, skipping the DataNode process entirely -- see short-circuit reads. Over HTTP every byte is read by the DataNode process and written to a socket, with HTTP framing and chunked encoding on top, and through a gateway there is a second full copy of the stream.

Second, no per-file parallelism. A native client can be draining block N from one DataNode while prefetching block N+1 from another. A WebHDFS read is a single HTTP response from a single node, and that node's NIC is your read bandwidth for the whole file -- remembering that it is itself fetching the blocks it does not hold.

Third, and most easily missed: placement. Writes go through the ordinary write pipeline, and the head of the pipeline is chosen relative to the writer, with a local DataNode preferred when there is one. For HttpFS that is a trap with a long fuse. Co-locate the gateway with a DataNode and the first replica of everything written through the gateway lands on that one node. It fills ahead of its peers, the balancer runs continuously chasing a skew that is being recreated faster than it is corrected, and the node becomes a read hotspot as well because it holds a replica of every recent file. Run HttpFS on a host with no DataNode on it, and the problem never exists.

Native WebHDFS from outside the cluster gets this right by accident: the client is not in the topology map, so there is no local node to prefer and targets spread across the cluster.

Failure modes worth rehearsing

Permission denied for user dr.who. No identity was sent, under simple authentication. Add user.name, or better, stop using simple authentication.

403 with StandbyException in the response body. You reached the standby NameNode. An HA-aware client retries the other one automatically; a hand-rolled curl script has to do it itself, and hardcoding one NameNode's hostname is a failover outage waiting for its trigger. See HDFS High Availability.

A redirect to a host the client cannot reach. Covered above, and worth repeating only as a test discipline: validate the endpoint from the network the clients actually run on, never from an edge node inside the cluster.

LISTSTATUS on an enormous directory. The NameNode builds one JSON document containing every entry and ships it in one response, which is a heap spike on the server and a parse spike on the client. LISTSTATUS_BATCH with a startAfter cursor pages through it instead. Use it for any directory whose size you do not control.

Zero-length files with no error. The most confusing WebHDFS report there is. A client library that transparently follows the 307 but drops the request body -- or drops the Authorization header when the redirect crosses to a different host -- can complete the write with a success status and an empty file. If your uploads arrive empty, stop debugging HDFS and check what your HTTP library does across redirects; noredirect=true plus an explicit second request removes the ambiguity entirely.

Length that never changes during a write. A file's length becomes visible when the stream is closed, so a poller watching GETFILESTATUS for progress on an in-flight write learns nothing. Track progress on the writing side.

Token expiry mid-transfer and thread pool exhaustion round out the list. Slow readers hold server threads for the duration of their stream, so a gateway sized for throughput can still fall over on concurrency -- and the symptom is queued or refused connections rather than slow ones, which sends people looking at the network instead of the pool.