YARN's central design move was splitting Hadoop 1's monolithic JobTracker into a cluster-wide ResourceManager and a per-application ApplicationMaster. The RM owns cluster capacity and queue policy; it deliberately knows nothing about map tasks, Spark stages, or Tez vertices. Everything application-specific — what work exists, what shape of container each unit of work needs, what to do when one dies — lives in the AM, a short-lived process that YARN launches inside a container just like any other. That is why the AM is simultaneously the most powerful and the most fragile component in a YARN job: it is the only piece that understands your application, and it is running on a commodity worker node that can disappear. This article walks the full AM lifecycle — how its container comes to exist, the three calls of ApplicationMasterProtocol, the heartbeat and allocation loop, resource requests with locality relaxation, launching task containers over ContainerManagementProtocol, AM liveness and expiry, attempt limits and work-preserving restart, unmanaged AM, clean shutdown with log aggregation, and how to diagnose an AM that has silently stopped making progress.
Why it matters
ApplicationMaster design determines how well an application handles failures. A well-designed AM detects failed task containers, requests replacements, and restarts them from checkpointed state. A poorly-designed one gives up on first failure or, worse, keeps requesting resources it cannot use. The quality of Spark or Tez as YARN citizens comes down largely to their AM quality.
Understanding AM behavior also helps diagnose stuck applications. If your job hangs at 99 percent completion, the AM is usually where the problem lives — either it is not requesting the last container correctly or it is not shutting down cleanly after work completes.
There is a second, less obvious reason to care: the AM is the only per-application process the cluster operator does not control. The RM, the NodeManagers, and the scheduler are all cluster software with cluster-wide tuning. The AM is your code (or your framework's) running with a resource allocation, an RPC channel to the scheduler, and a security token that lets it ask for more of the cluster. An AM that leaks container requests, heartbeats too aggressively, or refuses to release completed containers degrades the whole cluster, not just its own job. Most "the scheduler is broken" escalations turn out to be one badly behaved AM.
The AM is also where the boundary between framework and platform is drawn. YARN gives you allocation and isolation and nothing resembling a task model, a DAG, a retry policy, or a shuffle; every one of those is implemented in an AM. Reading one — MapReduce's MRAppMaster, Spark's ApplicationMaster, or the distributedshell example that ships with Hadoop — is the fastest way to see what YARN actually promises.
The architecture
The AM has two RPC endpoints it talks to. First is the ApplicationMasterProtocol, used to communicate with the RM: register, request containers, release containers, send heartbeats, and unregister. Second is the ContainerManagementProtocol, used to talk to NodeManagers: launch a container, stop a container, get container status.
Internally, an AM typically has a component that decides what tasks need to run (the planner), a component that requests containers based on task needs (the allocator), a component that launches tasks in allocated containers (the launcher), and a component that tracks task progress and handles failures (the monitor). The details differ per framework but this decomposition is universal.
Hadoop ships client libraries that implement the tedious half of both protocols. AMRMClient and its callback-driven sibling AMRMClientAsync manage registration, the outstanding-request table, and the heartbeat timer against the RM; NMClient and NMClientAsync manage the NodeManager side including token plumbing via NMTokenCache. Almost no production AM speaks the raw protobuf protocols directly, and you should not either — the request-accounting rules described later are easy to get subtly wrong by hand.
How it works end to end
The AM starts when the RM allocates its container and instructs an NM to launch it. Once running, the AM registers with the RM by sending an ApplicationMasterRegistrationRequest. It then enters its main loop: send resource requests describing needed containers (memory, vcores, node preferences), receive allocations from the RM, launch tasks in those allocations by calling the NM, and monitor progress.
When a task container completes, the AM either treats it as success and marks the corresponding work done, or treats it as failure and reschedules the task. Failure handling policy is entirely up to the AM: how many retries, whether to blacklist nodes that fail repeatedly, whether a certain failure percentage should fail the whole application.
When all work is done, the AM sends UnregisterApplicationMasterRequest to the RM and exits. The RM marks the application succeeded and reclaims the AM's container. If the AM crashes without unregistering, the RM detects timeout on the heartbeat and either restarts the AM (up to a retry limit) or fails the application.
Two details make that shutdown safer in practice. First, finishApplicationMaster() takes a FinalApplicationStatus (SUCCEEDED, FAILED, or KILLED), a diagnostics string, and a final tracking URL — that diagnostics string is what yarn application -status shows the user, so an AM that fails with an empty diagnostic has thrown away its own postmortem. Second, the call returns an isUnregistered flag; the AM must keep calling until it is true, because if the process exits before the RM has recorded the unregistration the attempt is scored as a crash and may be retried. A brief bounded retry before System.exit() is not optional politeness, it is correctness. The rest of this article expands each step of that loop.
From submission to a running AM container
An AM does not start itself. The client builds an ApplicationSubmissionContext and hands it to the RM over ApplicationClientProtocol. That context carries the application name, the target queue, the priority, the AM container's resource request (memory and vcores), and — most importantly — a ContainerLaunchContext describing exactly how to start the AM process: the command line, environment variables, local resources to stage in, and security tokens.
Before submitting, the client uploads the AM's dependencies (the job jar, configuration, any archives) to a staging directory in HDFS, typically under /user/<user>/.staging/<appId>. Those HDFS paths become LocalResource entries in the launch context. The NodeManager's localizer will later download them into the container's working directory, which is why an AM's classpath is expressed as relative paths inside the container rather than absolute cluster paths.
The RM's ApplicationsManager accepts the submission, assigns an ApplicationId, and creates the first ApplicationAttemptId — note the two-level identity, application_<ts>_<n> and appattempt_<ts>_<n>_<attempt>, which exists precisely because the AM can be restarted. The scheduler then allocates one container for the AM out of the target queue's capacity, and the RM instructs the owning NodeManager to launch it. For MapReduce that container is sized by yarn.app.mapreduce.am.resource.mb (default 1536) with heap flags in yarn.app.mapreduce.am.command-opts; Spark uses spark.yarn.am.memory in client mode and spark.driver.memory in cluster mode. The AM container consumes real queue capacity like any other container — a queue full of AMs and no room for task containers is a classic deadlock, which is what yarn.scheduler.capacity.maximum-am-resource-percent (default 0.1) exists to prevent.
Registering with the ResourceManager
The launched AM finds its identity from the environment: ApplicationConstants.Environment.CONTAINER_ID gives it the container it is running in, from which it derives its ApplicationAttemptId. Its credentials arrive in the container's token file, pointed at by HADOOP_TOKEN_FILE_LOCATION, and include the AMRMToken — the only credential that authorises calls on ApplicationMasterProtocol. The RM rolls the master key behind that token on a schedule governed by yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs, and the client library transparently picks up the rolled token from allocate responses.
The first thing a running AM must do is call registerApplicationMaster(host, rpcPort, trackingUrl). The host and port advertise the AM's own service endpoint (for MapReduce, where the job client polls for counters). The tracking URL is the application's web UI; the RM does not link to it directly but to the web application proxy (yarn.web-proxy.address), which strips the user's cluster credentials before forwarding — an untrusted AM must never be handed the browser's Hadoop cookies.
The RegisterApplicationMasterResponse is the AM's contract with the cluster and is routinely ignored by first-time implementers. It carries getMaximumResourceCapability(), the largest container the scheduler will ever grant (bounded by yarn.scheduler.maximum-allocation-mb and -vcores); the queue the application actually landed in after placement rules; getSchedulerResourceTypes(), which tells you whether vcores are even enforced; and getContainersFromPreviousAttempts(), which is non-empty only on a restarted attempt. Registering twice for the same attempt is an error, and calling allocate() before registering raises ApplicationMasterNotRegisteredException.
The allocate heartbeat: one call that does everything
After registration the AM has exactly one RPC it makes for the rest of its life: allocate(AllocateRequest). That single call is simultaneously the heartbeat, the resource request, the release channel, the blacklist update, and the progress report. An AllocateRequest carries a float progress value in [0,1], a list of ResourceRequest objects (the ask), a list of ContainerIds to release, a ResourceBlacklistRequest, and a monotonically increasing responseId.
The responseId is the deduplication mechanism. The RM keeps the last response per attempt; if the AM re-sends the same id, it gets the cached response back rather than double-counting the ask. If it sends an id the RM does not expect — because the AM restarted, or because a network retry replayed an old request — the RM throws InvalidApplicationMasterRequestException. Handling that correctly means re-registering, not retrying blindly.
The reply, AllocateResponse, contains newly allocated containers, statuses for containers completed since the last call, updated node reports, available headroom for the AM's queue, and any AM command such as AM_SHUTDOWN or the older AM_RESYNC. Headroom deserves attention: it says how much more the queue will give right now, and an AM that ignores it will hold outstanding requests the scheduler can never satisfy.
Heartbeat cadence is the AM's choice, but the ceiling is not. If the RM does not hear an allocate() within yarn.am.liveness-monitor.expiry-interval-ms (default 600000, ten minutes), checked every yarn.resourcemanager.amliveliness-monitor.interval-ms (default 1000), it declares the attempt dead and kills its container. MapReduce heartbeats every yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms (default 1000). Too fast and you overwhelm the RM's yarn.resourcemanager.scheduler.client.thread-count handler pool; too slow and you are one long GC pause away from expiry.
Resource requests, locality and relaxation
A ResourceRequest has five fields: a Priority, a resourceName, a Resource capability, a container count, and a relaxLocality flag. The counter-intuitive part is that YARN's ask is not a list of individual container requests — it is a declaration of current outstanding demand per (priority, resourceName, capability) triple. Sending "5 containers at priority 1 on ANY" twice does not request ten containers; it restates the same five. Frameworks that forget to decrement the ask as containers arrive are the ones that hold a queue hostage.
Locality is expressed by submitting the same logical request three times at three resourceName levels: a hostname, a rack path such as /rack1, and the wildcard ResourceRequest.ANY (*). The node- and rack-level entries are hints about where data lives; the ANY entry is the one that actually authorises the scheduler to hand you a container. Omit ANY and you will wait forever even though the scheduler has capacity — a genuinely common bug in hand-rolled AMs.
relaxLocality controls whether the scheduler may fall back. With it true (the default), the Capacity Scheduler waits a bounded number of missed scheduling opportunities — yarn.scheduler.capacity.node-locality-delay, default 40 — before satisfying a node-local request from elsewhere in the rack, then off-rack. Set it false and the request becomes a hard placement constraint, which is how services that must land on specific hosts are expressed. Richer placement (affinity, anti-affinity, cardinality) uses the newer SchedulingRequest API with PlacementConstraints, and coarse partitioning uses node labels.
One more surprise: the scheduler normalises every capability upward to a multiple of yarn.scheduler.minimum-allocation-mb (default 1024) and rejects anything above yarn.scheduler.maximum-allocation-mb with InvalidResourceRequestException. Ask for 1100 MB on a default cluster and you are charged for 2048.
Launching and supervising task containers
An allocated Container is a lease, not a running process. It carries the target NodeId, the granted Resource, and a container token that proves to the NodeManager that the RM really did authorise this allocation with these limits. The AM must now open a connection to that specific NodeManager and call startContainers() on ContainerManagementProtocol, passing a ContainerLaunchContext exactly like the one the client built for the AM itself: command line, environment, local resources, and credentials. Authentication uses a per-node NMToken, cached by NMTokenCache and refreshed through allocate responses.
The lease expires. If the AM does not launch an allocated container within yarn.resourcemanager.rm.container-allocation.expiry-interval-ms (default 600000), the RM reclaims it and the AM sees it as completed with an expiry diagnostic. An AM that allocates in bulk and launches slowly — for example one blocking on HDFS while holding fifty allocations — will bleed containers this way and appear to be starved by the scheduler when it is really starving itself.
Once launched, the AM supervises. It can poll getContainerStatuses(), but the authoritative signal is AllocateResponse.getCompletedContainersStatuses(), which reports each finished container's exit status and diagnostics. Exit code 0 is success; 143 is SIGTERM, usually a memory kill by the NodeManager or a preemption; ABORTED (-100) means the node was lost or the container was preempted; DISKS_FAILED (-101) means the NM's health checker failed the volumes underneath it. Reading those correctly is the difference between retrying a task and pointlessly retrying it on the same bad host — which is what the ResourceBlacklistRequest in the next allocate call is for, driven in MapReduce by yarn.app.mapreduce.am.job.node-blacklisting.enable. Whether a slow container should be duplicated rather than replaced is a separate policy, covered in speculative execution.
Failure, attempts and work-preserving restart
When an AM dies — crash, OOM kill, expiry, or lost node — the RM starts a new attempt: same ApplicationId, incremented ApplicationAttemptId. How many times it will do that is min(yarn.resourcemanager.am.max-attempts, ApplicationSubmissionContext.getMaxAppAttempts()). The cluster-wide key defaults to 2 and is a hard cap: an application asking for 10 attempts on a cluster configured for 2 gets 2. Spark exposes the per-app half as spark.yarn.maxAppAttempts, MapReduce as mapreduce.am.max-attempts.
For long-running applications a flat attempt budget is wrong — a streaming job running for three months will legitimately lose its AM more than twice over that window without being broken. setAttemptFailuresValidityInterval() makes the budget a sliding window instead: failures older than the interval no longer count. Spark surfaces it as spark.yarn.am.attemptFailuresValidityInterval; setting it to something like one hour is standard for Spark Streaming and Flink session clusters.
By default a new attempt starts from nothing: the RM kills every container the previous attempt held. setKeepContainersAcrossApplicationAttempts(true) changes that — the running task containers survive, and the new AM finds them in RegisterApplicationMasterResponse.getContainersFromPreviousAttempts(). This is only useful if the AM can genuinely re-adopt work in progress, which requires it to have externalised enough state (typically to HDFS or ZooKeeper) to know what those containers were doing. Spark uses this for dynamic allocation recovery; Flink uses it to keep TaskManagers alive across JobManager restarts.
Restart also depends on cluster-side recovery. yarn.resourcemanager.work-preserving-recovery.enabled (default true) lets the RM itself restart without killing running applications, backed by a state store such as ZKRMStateStore via yarn.resourcemanager.store.class. On the worker side, yarn.nodemanager.recovery.enabled (default false, and worth enabling) plus yarn.nodemanager.recovery.dir let a NodeManager restart while its containers keep running. Without NM recovery, a rolling NodeManager upgrade kills every container on every node, AMs included, and your attempt budget evaporates for reasons unrelated to your code.
Unmanaged AM and other deployment modes
An unmanaged AM inverts the launch path. Setting ApplicationSubmissionContext.setUnmanagedAM(true) tells the RM not to allocate or launch an AM container at all; instead the RM registers the attempt, mints an AMRMToken, and hands it back to the client, which runs the AM process itself — usually on an edge node, outside the cluster's resource accounting. Hadoop ships UnmanagedAMLauncher in hadoop-yarn-applications-unmanaged-am-launcher as a reference driver.
The trade is explicit. You gain a debuggable AM: it runs in your terminal or your IDE, you can attach a profiler, and its stdout is right there rather than buried in aggregated logs. You lose everything YARN normally provides — no resource isolation for the AM, no automatic restart on failure (there is no container for the RM to relaunch), and no accounting of the memory it consumes. It is a development and integration tool, and the basis for out-of-cluster schedulers, not a production deployment mode.
The managed case still has two meaningfully different shapes, best illustrated by Spark. In cluster mode the driver is the AM: it runs inside the container, survives client disconnection, and owns both planning and allocation. In client mode the driver stays on the submitting host and the AM degrades to a thin "ExecutorLauncher" that does nothing but negotiate containers on the driver's behalf — which is why killing your laptop's spark-shell kills the job in client mode but not in cluster mode. Long-running services (Flink session clusters, YARN Services / the former Slider) take the pattern further: the AM becomes a supervisor that never finishes, continuously replacing failed containers and re-registering after its own restarts.
Debugging an AM that is stuck
Start by separating "not scheduled" from "scheduled but stuck". yarn application -status <appId> shows the state: ACCEPTED with no progress means the AM container itself has not started — the queue is full, the AM resource percent cap is hit, or the requested AM container exceeds yarn.scheduler.maximum-allocation-mb. RUNNING at a frozen progress value means the AM is alive and failing to make headway. yarn applicationattempt -list <appId> tells you how much of the attempt budget you have already burned, which is often the first sign that the job has been quietly crash-looping.
For a running AM, the RM web UI's application page shows the outstanding ask broken down by priority and resource name, plus current headroom. Two patterns dominate. Outstanding requests with zero headroom is a capacity problem, not an AM problem — go look at the queue. Outstanding requests with headroom is almost always a request-accounting bug: an ask registered at node and rack level but never at ANY, a capability above the maximum allocation, or an ask the AM forgot to decrement after the container arrived.
The AM's own logs are the next stop. yarn logs -applicationId <appId> -am 1 fetches the first attempt's AM log specifically, which beats downloading gigabytes of task logs. That only works if aggregation is on: yarn.log-aggregation-enable=true, retained for yarn.log-aggregation.retain-seconds. For a long-running AM whose logs you need while it runs, set yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds (3600 is typical) so partial logs upload on a rolling basis instead of only at container exit.
Check the boring failure modes before the interesting ones. An AM killed at exactly the ten-minute mark with no error is expiry — a GC pause or a blocked heartbeat thread, not a scheduler fault. An AM killed with exit 143 and a "running beyond physical memory limits" diagnostic needs a bigger AM container, not a bigger heap flag: on a large MapReduce job the AM tracks state for every task, and the default 1536 MB runs out past a few tens of thousands of them.