New MIT research overturns prior view about how AI capabilities could overtake human workers

By Rachel Gordon and MIT FutureTech

Anthropic CEO Dario Amodei has said that AI could surpass “almost all humans at almost everything” shortly after 2027. While AI’s capabilities are certainly improving, such rapid progress might seem at odds with findings that show AI is still failing at 95%+ of remote freelance projects, and continues to struggle with hallucination, long term planning, and forms of abstract reasoning that humans find easy. But recent work from METR has found evidence that LLMs can gain capabilities in rapid surges — jumping from succeeding almost never to almost always in just a few years. If this is true across the economy, it could mean that workers could be blindsided by AI advances.

In their study, MIT researchers characterize these out-of-nowhere capability gains as “crashing waves” and ask if they are likely to be an economy-wide phenomenon or whether advances in AI come as a “rising tide”. Across thousands of real world tasks, the team finds that, while indeed AI capabilities are improving quickly, AI capabilities are rising more smoothly, suggesting that “crashing waves” are the exception, not the rule.

The MIT researchers find:

• AI performance is rising smoothly across tasks in many parts of the economy and across very different task durations, suggesting that AI capabilities are arising tide. “This isn’t inherently protective for workers, tides could still rise quickly, but it does suggest that workers and policymakers monitoring progress should be able to see AI improvement coming," says senior author Neil Thompson.
• AI capabilities are already strong. The researchers focused on the 63% of tasks that workers in the US economy do that are text-based, and therefore could potentially be done by LLMs. Amongst these, when given the right information, LLMs were able to complete 60% of the tasks they were given at a level that a manager would describe as “minimally sufficient” without human involvement. Only 26% were of “superior” quality. Said lead researcher Matthias Mertens “LLMs demonstrated impressive proficiency, even on their own”.
• AI capabilities are rising quickly. While the MIT study finds that 2027 is too aggressive an estimate for AI to broadly eclipse the performance of human workers, it still finds rapid progress. Their projections suggest that AI will achieve 80% success rates on most tasks by 2029. Although as Dr. Thompson stressed “these depend on continued progress in AI hardware and algorithms and scaling of AI models. If these slow, so will the pace of AI capability increase.”
• The study’s findings have important impacts for policymakers and businesses trying to prepare for the coming changes brought about by AI.

You can read the full paper on the FutureTech website.

This post was originally published by MIT CSAIL and is republished here with permission.

Reviewed by Asim BN.

Read next:

• Why AI Leaders Are So Focused On Image Generation

• Think different — for 50 years

by External Contributor via Digital Information World