Note: This is the 6th part of a series looking at how people and companies and governmental organizations are responding to the impact of widespread implementation of AI on employment in the US. You can find the other parts beginning here.
Here’s something I don’t think I’ve ever heard before. There’s a growing chorus of voices at cutting-edge private companies pleading with the government: PLEASE regulate us.
It’s been a crazy few weeks in the AI world. In July, 1,386 researchers from a series of leading AI companies signed a letter calling on the US government to support an international effort to “develop the technical and governance tools needed to deliberately pace (note: in techspeak, “pace” means “slow down”) the frontier of automated AI development.”
This week things got more intense when a guy named Jacob Coxon announced his resignation from Anthropic. Coxon, who’s spent the past three years building models for both OpenAI and Anthropic outlined the reasons for his resignation in a series of tweets. Because of the attention the tweets got, it’s worth quoting from a few of them at length:
“Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight and acquire real power and resources.”
“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.
“A common response is ‘if they truly believe this, why are they still building it?’ At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well understood, but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves, despite the risk.”
By midday Friday, September 11, less than three days after he posted them, Coxon’s tweets had been viewed by 165 million people and his overall assessment was validated on X by a current safety researcher at Anthropic, Evan Hubinger: “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” Geoffrey Irving, another former Anthropic and OpenAI employee and former chief scientist at the UK’S AI Security Institute, upped the ante still further on X on Wednesday: “I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years.”
Josh Engels, former head of safety research at Google, added this in an interview with NBC News: “There are no adults in the room. People are trying their best but there is no one coming to save us.”
Josh Engels: “There are no adults in the room.”
Finally, on Saturday, Anthropic CEO Dario Amodei published an essay called “We Must Pace the Frontier,” arguing that AI companies should voluntarily slow their development of new models, adopt shared safety standards and bring in “embedded evaluators”to make sure nobody was cheating, while waiting for governments to set official rules and regulations. Sam Altman, Elon Musk and Demis Hassabis, chair of Google’s DeepMind, all joined in. On Face the Nation Sunday, Amodei went farther, explicitly endorsing regulation: “Regulation constrains the private companies. Regulation allows the public and its elected representatives to have a say, and limits what the private companies can do.”
All these folks are asking for government to play a very active role.
But so far, while the people who know AI best are describing a meteor heading toward the earth, government’s response has been to pick up a flyswatter.
· China and the state of California have put some age restrictions on the use of chatbots.
· The EU has required AI companies to report on serious incidents of AI agents breaking out of their boxes; Congress has introduced a similar bipartisan bill.
· A member of Parliament in Britain has introduced a bill banning superintelligence altogether. A group of Democrats and Independent Senator Bernie Sanders have called for a pause in AI development. Neither proposal has advanced.
· US House Speaker Mike Johnson has suggested he might call AI companies together to reach “consensus” on safety guardrails – maybe sometime this winter.
· The Trump administration has a screening mechanism to review new AI models, but doesn’t require any reporting of real world incidents.
The people who know most about this technology are asking for a different level of help. They believe AI is becoming too powerful too fast. They won’t unilaterally disarm, either because they don’t want to miss out on the huge potential revenue AI will bring or because they think only they can act responsibly with it. But they know they can’t fix the problems on their own. And the only way “pacing” – deliberate slowing down – can work is if the power players in all the countries of the world agree to do it at the same time.
If the idea of a global compact to stop a technology that could wipe out the world sounds impossible – well, we’ve done it before. Back in 1968 it was clear that both the US and the Soviet Union had enough nuclear weapons to destroy the world and, in their ever-escalating quest to achieve “mutually assured destruction” (the appropriate acronym was MAD) they were continuing to build more weapons at an absurdly expensive, ever-more-dangerous pace. Instead they jumped out of the race and developed a Nonproliferation Treaty (NPT) capping the sharing of nuclear technology with nonnuclear states. The agreement never stopped the development of nuclear weapons, but it pushed a pause button, and gave people the time they needed to lay out a shared set of rules for the development of weapons.
The US and the USSR put the first signatures on the NPT in 1968; it didn’t stop the MADness, but it did slow it down.
In the case of the NPT, the essential players were the US and the USSR – once they signed on, eventually 191 other countries joined. In the case of an international AI nonproliferation treaty, it would be the biggest players in developing AI – the US and China.
One challenge to movement toward an AI treaty is that the threat of nuclear weapons was easily visible to everyone. The potential extinction threat AI poses is so far only visible to a small number of people operating in secrecy. That said, this week’s activity shows that those fears are finally gaining serious attention, and there has been good work done on what an AI treaty might look like, led by a group called the Machine Intelligence Research Institute, or MIRI.
Instead of tracking supplies and shipments of uranium, the AI treaty could track the supply of the ultra-fast chips needed to reach artificial superintelligence. There would need to be an international monitoring agency similar to the International Atomic Energy Agency with authority to inspect research sites. Putting those kinds of mechanisms in place might slow things down just enough that we could understand what “artificial superintelligence” is and how we might govern the machines that develop it.
Both sides need to push the pause button at the same time (Image generated with DeepAI.org).
China appears open to some sort of agreement. In July, Chinese President Xi Jinping called for a “consensus-based global governance framework” and for “laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use, and ensure that AI is always under human control.” A South Korean online source claims that China and the US have been having preliminary talks about some kind of agreement.
President Trump and President Xi are meeting in late September, with AI safety supposedly on the agenda. Up until now, President Trump has been reluctant to put any restrictions on AI development, and this weekend Trump’s former AI czar David Sacks basically told AI companies not to involve government, but to police themselves. But perhaps now, with the encouragement of every CEO of every major US-based AI company calling for a slowdown, the Trump administration’s position will change.
It will likely be incredibly difficult to work out details of an agreement like that. But if we are to believe the people who know most about where AI is headed, we should encourage the leaders to try anyway.
Notes:
“Pacing” letter: https://www.pacingthefrontier.com
Benton and Enger on importance of controls on AI: https://www.nbcnews.com/tech/security/two-ai-researchers-leave-anthropic-google-safety-concerns-rcna597086
Coxon’s posts on X: https://x.com/hilbertspaess/status/2097476196791709843
Background on Hubinger’s tweet: https://www.bbc.com/news/articles/ckgwy1k42w4o
Summary of efforts by government to begin regulation of AI: https://www.axios.com/2026/09/11/ai-safety-debate-next
Potential AI nonproliferation treaty: https://intelligence.org/2026/05/12/summary-an-international-agreement-to-prevent-the-premature-creation-of-artificial-superintelligence/
Background on nuclear nonproliferation agreement of 1968: https://history.state.gov/milestones/1961-1968/npthttps://en.wikipedia.org/wiki/Treaty_on_the_Non-Proliferation_of_Nuclear_Weapons
MIRI’s outline of a potential AI treaty: https://intelligence.org/2026/05/12/summary-an-international-agreement-to-prevent-the-premature-creation-of-artificial-superintelligence/
President Xi on AI safeguards at World AI Forum: http://english.scio.gov.cn/topnews/2026-07/18/content_118605932.html