Why it matters
Straggler tasks are the number one cause of job long-tail latency in real clusters. A cluster where 99 percent of tasks finish in an hour but one percent take three hours has an average job runtime of three hours. Speculation cuts that tail dramatically without requiring any user code changes.
The cost is real but usually acceptable: extra CPU on nodes that would have been idle anyway. On very tight clusters where every CPU is contested, disabling speculation is defensible.
The architecture
Speculation runs in the ApplicationMaster. The AM tracks progress for every task and computes a progress rate (percent completed per unit time). Tasks whose progress rate is significantly below the average are candidates for speculation. When a candidate has been running for a minimum duration (to avoid speculating too eagerly), the AM requests a duplicate container from the RM and launches a second instance on a different NM.
Both instances continue running. The framework treats whichever one completes first as the authoritative output. When the first instance finishes, the AM kills the other, discarding its partial work.
How it works end to end
Progress rate calculation depends on the task type. For map tasks, progress is defined as the fraction of input bytes read. For reduce tasks, progress is a weighted combination of shuffle progress, sort progress, and reduce progress, each contributing one third of the total. The AM samples progress on each heartbeat and updates its running average.
When a task falls below a threshold (typically 25 percent behind the average progress rate) and has been running for at least a minute, the AM adds it to the speculative candidate list. Only one speculative copy runs per task at a time. If both original and speculative are equally slow, no third copy is launched.
Output handling is critical. Both instances write to their own staging directory. When the AM chooses a winner, only the winner's staging directory gets renamed to the final output location. The loser's staging is deleted, so partial output never becomes visible.