The capability numbers that drive AI purchasing come from a narrow slice of work. Three pieces that surfaced this week measure how narrow, and what happens when tests get closer to real jobs.
Mercor's benchmark map
Mercor's "US Benchmark Map", discussed on Hacker News on October 16 with about 600 points and 540 comments, maps 29 public, agentic, unsaturated benchmarks onto the US Department of Labor's O*NET taxonomy of 923 occupations and 18,797 tasks. An LLM judge tags each benchmark prompt with matching tasks, and a task counts as covered only with at least 5 prompts. Result: "only 208 of 923 occupations have a task with at least 5 benchmark prompts, and 6 of 22 job families have no such occupations." Combining coverage with estimated time shares, Mercor concludes that "90% of American working hours are spent on tasks with little or no coverage from existing benchmarks."
Coverage follows money. Benchmarks "heavily cover AI-exposed lawyers' tasks, but barely cover any of the O*NET tasks of similarly AI-exposed legal secretaries or court clerks." Mercor is careful about limits: the list is incomplete, which "will necessarily bias down estimates", and "these results are provisional." Mercor's own APEX benchmarks are part of the index.
FDE-Gym: the work around the task
FDE-Gym (arXiv, October 15) builds 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, adapted by industry experts from real projects. Agents must talk to scripted stakeholders and change production-like systems, and they are scored on improvement to the customer's own success metric, with penalties for each message, trial run and wait. Across 13 configurations from seven providers, the best "achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks." The authors name the bottleneck: "Discovering unstated constraints is the primary bottleneck". Even on the 23 tasks where the best configuration obtained every key fact, it reached a high business uplift on only 10.
OpenGCUA: the harness is part of the score
OpenGCUA (arXiv, October 15) is an open stack for computer-use agents that mixes GUI actions with code execution and moves some decisions from fixed harness rules to the model. On OSWorld-v2 its full configuration improved scores for all eight models tested, including GPT-6 Astra and Claude Opus 5, raising mean partial score by 19.6 points and cutting steps per task by 53.0% compared with GUI-only execution. On four real macOS tasks, Qwen3.8-Max runs used 19.4 times as many reasoning tokens as GPT-6 Astra.
I think the three pieces describe one measurement problem. Benchmarks cover the well-specified, high-wage slice of work. Real jobs are mostly discovering what was not specified. And the scaffolding around a model can move a score by double digits. A leaderboard number therefore says little about whether an agent will do your job, done your way, in your harness.
What remains uncertain. Mercor's 90% depends on LLM tagging and LLM-estimated time shares. FDE-Gym's stakeholders are scripted, not real customers. OpenGCUA's gains are its authors' measurements. None of the three has been independently replicated.
Analysis: For teams buying or building agents, the actionable step is the one none of these papers can do for you: build a small private eval from your own tickets, including the ambiguous ones, and score it on the outcome the business cares about, run through the harness you will actually deploy. Public benchmarks remain useful for ranking models on what they cover. Most of the work does not appear in them.
Mercor's benchmark map finds only 208 of 923 US occupations have any task covered by 5+ prompts across 29 agentic benchmarks and estimates 90% of US work hours are barely benchmarked (provisional); FDE-Gym's best agent resolves 14 of 95 real delivery tasks, failing mainly at discovering unstated constraints; and OpenGCUA shows harness design lifting all eight models' OSWorld-v2 scores by 19.6 points on average.