Mercor's Benchmark Map finds AI tests cover little of the US work that takes up most hours
- Mercor's Benchmark Map counts 29 AI benchmarks against 18,797 tasks, and only 208 of 923 occupations have a task with at least 5 benchmark prompts.
- Mercor estimates that 90% of American working hours go to tasks with little or no benchmark coverage, which makes it hard to measure how well AI handles real jobs.
- Coverage skews toward high wage bill occupations, so benchmarks heavily cover lawyers' tasks while legal secretaries and court clerks get almost none.
- 6 of the 22 O*NET job families have no occupation with at least 5 benchmark prompts, and Personal Care has 0 of 31 covered.
- The benchmarks were chosen for public availability, agentic tasks, and economic realism, and Mercor's APEX benchmarks are included through held-out prompts.
Hacker News opinions
A lot of what humans do has no tight feedback loop, and agentic training needs one. That's why I don't buy the jobpocalypse yet.
We're on a forum full of software engineers, so we might be overestimating how fast this hits everyone's job, not just code.
I don't know if it's 90%, but a lot of what I do never shows up in tracking until I'm gone and things start falling apart.
About 80% of my software job is fighting stakeholder disagreements. Concurrent model sessions just give up control and pick a decision fast, which humans never do.
Stakeholder is basically the top corporate bullshit role. It lets a group exist so nobody can be blamed for excluding them, and it gives useless feedback so the group looks like it cares.
The next while will show leadership how little of getting something to prod is actual coding. Features don't ship the same day they're imagined.
Most of the cost savings from AI in software come from labor inefficiency, not the tech. It's a human capital problem more than a technology problem.
I think this mixes up correlation and causation. Verifiable digital tasks are easier to hill-climb on, and white collar jobs just happen to pay more.
Customer support will get decimated by AI, but those aren't highly paid jobs. Robots doing the dishes is a separate question.
They don't really want anything. They push enterprises with fear of missing out, and the cost-metric obsession is an illusion for many companies that jump on board.