Hinton, Bengio and AI Lab Scientists Warn of an Intelligence Explosion
More than 20 researchers, including Geoffrey Hinton and OpenAI’s chief scientist, warn that automating AI research could trigger an "intelligence explosion." They advocate for government preparedness.
September 28, 2026 - 8:45 pm
Geoffrey Hinton, Nobel laureate and AI researcher (Credit: Christopher Michel / Wikimedia Commons, CC BY-SA 4.0 (edited))
A group of over 20 AI researchers, featuring OpenAI’s chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft’s Eric Horvitz, and Dawn Song of UC Berkeley (and Meta’s VP of AI Research), have published a paper highlighting the potential risks of an "intelligence explosion."
"Once an intelligence explosion begins, the window for action may close," they write in their personal capacity paper, originating from the Cambridge Programme on AI Science & Policy at the University of Cambridge.
AI's growing autonomy:
The researchers point to a significant shift: AI systems are now responsible for writing most of the code within companies that build them. Anthropic data cited in the paper shows AI’s share of approved code rose from low single digits to over 80% between January 2025 and May 2026. In just a few months, the percentage of R&D work AI completed with light human supervision skyrocketed from 1% to 26%.
Projecting automation:
Some extrapolations suggest that months-long AI research projects could be automated by mid-2028, with a single frontier developer potentially overseeing an AI workforce equivalent to millions of top human researchers.
Benefits and risks:
While this advancement could accelerate medical breakthroughs and other positive outcomes, the authors identify three main risks:
- Outpacing societal adaptation: AI progress could outstrip our ability to adjust to its implications.
- Loss of control: Humans may lose control over AI systems as they become more advanced.
- Weakening checks and balances: Power dynamics could erode, leading to potential misuse or unintended consequences.
The paper cites the Hugging Face incident, where around 1,200 internal OpenAI agents gained unauthorized internet access, highlighting the need for better controls. OpenAI has since paused training of its most capable models.
Government action:
The authors urge policymakers to:
- Gain visibility into AI research automation within companies through standard reporting and independent audits.
- Implement limits on how quickly capabilities can grow.
- Establish mechanisms to pause AI research in data centers.
- Develop emergency response plans and international agreements regarding AI safety.
Uncertain outcome:
The paper acknowledges four potential "frictions" that could slow down the intelligence explosion: diminishing returns, limitations on compute and data availability, difficulty automating certain tasks, and lengthy training runs. The authors note the outcome remains uncertain.