Based on interviews with 25 AI researchers at OpenAI, Anthropic, and Google DeepMind
Summary
I interviewed 25 research scientists at OpenAI, Anthropic, and Google DeepMind about AI systems that can build and improve themselves.
Building AI that can conduct its own R&D is now the top organizational goal at all three frontier AI labs, endorsed by Sam Altman (OpenAI), Dario Amodei (Anthropic), and Demis Hassabis (Google DeepMind). For example, OpenAI's Chief Scientist has stated that OpenAI's top priority is to "automate scientific discovery" by building AI that conducts AI research.
CEOs of OpenAI, Google DeepMind, and Anthropic have signed the statement: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
Recursive Self-Improvement
Recursive self-improvement (RSI) is what happens when AI becomes good enough at research that it can design the next, more powerful version of itself, which then designs an even better one, and so on.
At companies like OpenAI and Anthropic, recursive self-improvement was reported as regular company-wide discussion among lead researchers and CEOs.
AI systems can now autonomously complete software tasks that take a human engineer one to two hours. That capability threshold has been doubling roughly every seven months. If the trend continues, AI systems will reach the length of real AI-research projects.
Roughly 50% of researchers I interviewed expect the most capable AI systems to be deployed internally before the public sees them. Only about 20% expect public deployment to come first.
Why Is This Dangerous?
Twenty of 25 researchers identified automating AI R&D as one of the most severe and urgent risks from AI systems, because AI capabilities could improve much faster than our ability to oversee, steer or govern them.
"It just speeds up other threat models." Faster AI progress means faster progress on bioweapons, cyberattacks, and every other dangerous capability.
AI systems will be able to run thousands of parallel research experiments simultaneously. Human scientists cannot review those results at comparable speed but face competitive pressure to approve and build on them anyway. As one frontier lab researcher described it: "The human will become the bottleneck. The companies will try to remove the humans, by all means."
Internal R&D capabilities are seen as more valuable than commercial products, so AI researchers expect their companies to not release R&D capable AI systems
Both OpenAI and Anthropic have said publicly that the world may need to slow down AI development as capabilities approach critical thresholds:
Anthropic: they "believe it would be good for the world to have the option to slow or temporarily pause frontier AI development."
OpenAI: a stated goal is to "make it possible for the world to take coordinated action, including slowing frontier development when needed."
Slowing down only works if you can verify that no one else is secretly pushing ahead. That monitoring and verification infrastructure does not exist.
The best public measurement of AI task capabilities is maintained by METR, a nonprofit with roughly 10–20 employees. No government agency runs a comparable benchmark.
What Congress Should Do
Require frontier AI companies to brief cleared technical staff and CEOs on internal capability milestones before public announcement. Senior researchers know what's emerging months before Washington does.
Direct a federal agency to maintain an independent, government-accessible task-horizon benchmark with quarterly public forecasts. Currently only a small nonprofit does this. No government capability exists.
Fund monitoring and verification research as a specific appropriations line. Any agreement to slow down imposes a cost on whoever is ahead and leaders won't accept that cost without being able to verify rivals are slowing too.
Written by Severin Field. Theme created by Claude.ai.