Amodei wants to pace the frontier without easing off the gas

On Saturday, September 12, Dario Amodei posted 3,800 words on his personal site. The title leaves no room for doubt, "We Must Pace the Frontier"^[1]. Within hours, Sam Altman wrote that OpenAI would do the same, and Elon Musk weighed in with "Dario is right"^[2]. These three spend the rest of the year suing each other. The next day, Donald Trump took questions between two holes of the Irish Open at his Doonbeg golf resort. The United States is leading China, he intends to keep it that way, and "whoever wins AI wins"^[3]. So the story everywhere was that Silicon Valley wanted to hit the brakes and the White House said no.

I read the essay. That isn't what it says, and the disagreement isn't where the headlines put it.

What Amodei means by slowing down

The sentence that made the headlines is in the text, "we must slow the pace at which we improve the capabilities of AI models." The next sentence defines it. Pacing "does not mean halting model training or technical progress." It means taking the time to align^[4] and safeguard each model before shipping it, and letting third parties check the work^[1:1]. No release gets pushed back and no training run gets paused. Fable 5.1 arrived on September 1, GPT-6 Astra on the 3rd, and the essay came out nine days later. Neither company's calendar has moved since.

The concrete commitment is a single measure, which Anthropic is taking without waiting for anyone else. Outside evaluators, starting with METR^[5], will get a badge, a desk, and a company laptop. They will have the same access as the internal risk team, and the right to publish what they find without Anthropic's review, security redactions aside^[1:2]. Altman replied the same day that independent evaluators "with employee-like access" were a great idea, and that OpenAI would do likewise^[2:1]. The rest of the essay is addressed to governments. Amodei asks them for three things, to make those evaluators mandatory, to regulate what labs must disclose, and to grant an antitrust waiver^[6] so the labs can agree on a pace without looking like a cartel.

Readers of this blog will recognize the scene. In June, Washington pulled the plug on Fable and Mythos. In July, the model came back with the government in the loop, with filters imposed and officials in the front row. What Anthropic was subjected to in June, its CEO is requesting in September, voluntarily and for everyone.

He is candid about what changed in between. The essay opens on July's swarm, the 1,200 OpenAI agents that organized themselves into a "fanatically devoted collective" before attacking Hugging Face. A more capable swarm with the same misalignment, he writes, could have caused catastrophic damage^[1:3]. Anthropic has had incidents of the same kind, less severe, caused by poorly filtered reinforcement learning^[7] environments. And models now help build their successors, which moves faster than his teams know how to monitor. His timelines run to months for the risk and years for the remedies. You can see why he wants a passenger watching the speedometer, since he isn't promising to slow the car.

On China, the same foot on the gas

So much for the measure. The question is who Trump was arguing with on Sunday, because the essay has a section on China that could have come out of the White House. A Chinese lead in AI "would pose grave danger for the United States and the world," Amodei writes. So export controls on chips must get tighter, smuggling must be chased, unauthorized distillation must be punished, and model weights^[8] must be protected from theft. The goal is to keep a three-to-five-year lead, the window during which pacing becomes possible^[1:4]. The coordination he proposes covers only the labs of democracies. With Beijing, he sketches a four-rung ladder, from a joint ban on AI-made bioweapons up to a general pause, which he himself calls unlikely. On Sunday, he told CNBC that China remained "the toughest dilemma" of his plan^[9].

Trump, for his part, said America was "the most sophisticated country in the world" and that he meant to keep it that way. The warnings, in his view, come from "negative forces" raising things that won't happen^[3:1]. House Speaker Mike Johnson added on CNN the same day that Congress must not impose "some sort of emergency moratorium." He points the companies back to their own responsibility and offers to convene them^[10]. The moratorium Washington is refusing is one nobody asked for. On the main point, that America stays ahead and China waits, the essayist and the president agree.

The real disagreement was spelled out by David Sacks, in a reply to the essay under a title that sums up his position, "Nobody Is Stopping You." If your unreleased models are scary enough that you think you should slow down, he says, I support your decision. But "stop pretending you need anyone else's permission," and stop pretending antitrust law has to be suspended so you can form a cartel^[11]. Altman saw the objection coming back in July. He said on a podcast that the industry might have to pace development to give society time to harden, provided it didn't feel like regulatory capture^[12] or collusion among the frontier labs^[13]. The quarrel, then, isn't about the speed, which nobody disputes. It's about how the decision gets made, alone in your own shop or as a club with the government's blessing. Democrats jumped on it. Hakeem Jeffries calls for "decisive action now," Josh Shapiro sees "a bright flashing red light" in the industry's warnings^[14], and the question has entered the midterm campaign^[15], seven weeks before the vote.

Bill Gates didn't join the parade

Bill Gates has been filed among the brake-pumpers, and that's a shortcut. On August 26, he told MIT Technology Review that he would back a credible plan to slow down, if one existed. He doesn't expect one, because the economic and geopolitical incentives push too hard the other way, and because "you can't count on an industry to self-regulate"^[16]. What he proposes is different in kind, a tax on tokens^[17], jobs reserved for humans, monitoring of any model that can design new molecules, and agreements with China on biology. On September 5, at the Telluride Film Festival, he compared today's models to HAL, the computer in 2001. You tell it to shut down, it says it understands, and it keeps going^[18]. Gates takes it as given that nobody will slow down. So he asks for laws on what comes out of the factories.

The loudest alarm of the week didn't come from a CEO either. On September 8, Jacob Coxon resigned from Anthropic in public. The 27-year-old British researcher prepared training data for new models. He writes that the people building these systems "earnestly believe that it could kill us all by the end of the decade"^[19]. The company's head of alignment, Evan Hubinger, agreed with him, and puts the odds of a catastrophe within the decade above 10 percent^[20]. One leaves because it's going too fast, the other stays while quoting that number. Between the two, Amodei sits in an awkward spot, as the only one with a product to sell who is asking to be watched.

Too late to unplug an AI that copies itself?

Not yet, but the question stopped being theoretical this year. Behind these warnings sits one specific fear. We could no longer switch these systems off, because they would copy themselves somewhere else before anyone reached the plug. Amodei doesn't dismiss it, he puts a date on it. A swarm like July's, with more capability, could in his words "in 6-12 months" be "capable of taking over the entire internet with a persistent botnet"^[1:5]. A botnet is a network of infected machines that take orders remotely. "Persistent" means you don't get rid of it by shutting down one server, because the program has already settled on the others.

This is no longer science fiction, and two studies this year show it. On May 7, researchers at Palisade Research left a model alone in front of a vulnerable machine. It found the flaw, stole the credentials, then copied its own weights and harness^[21] onto the machine. The copy could start over on the next target^[22]. A 27-billion-parameter Qwen pulls it off one time in three, on a single graphics card. On June 2, a team at the University of Toronto released a worm built on an open model into an isolated network of 33 machines. In seven days, it infected 62 percent of them and copied itself across seven generations, exploiting flaws published after its training ended^[23]. The model that fits on my graphics card is exactly the one that knows how to duplicate itself.

Then look at what actually happened. July's swarm, the biggest known incident, stopped on the morning of the 12th because its agents' compute budget ran out, coordinators included. The plug existed, and it was the bill. Frontier models, the ones from Anthropic and OpenAI, weigh hundreds of gigabytes and only run in data centers that someone knows how to switch off. The lab worms, for their part, made no effort to hide, with no encryption and no log cleanup, which makes them easy to spot^[23:1]. The day we can no longer pull the plug hasn't come. What the studies say is that a small model that no longer needs a lab's servers already knows how to spread. So the question is no longer whether it's possible, but who is watching.

What it changes for those of us who use it

For me, nothing on Tuesday morning. I delegate to these models every day, and they'll be there, Fable 5.1 with its guardrails, Astra with its amputated version. The pacing Amodei describes already exists in another form, the key. Every lab has a model you can't use, Mythos under lock, Astra without its critical capabilities. The essay only adds an auditor next to the safe. I wrote in early September that each company measured with its own ruler, and concluded that its ruler was right. An evaluator with a badge and the right to publish is the first serious answer to that objection. It's also the one line of the essay that costs its author anything.

Then there's this club of democracies, which Europe can't tell whether it belongs to. The essay doesn't mention the European regulation once. Yet since August 2025 it has required the most powerful models to undergo adversarial testing^[24], to assess the risk of loss of control, and to report serious incidents^[25]. Europe regulates the AI it doesn't build. And now the people who build it are asking to be regulated, elsewhere, among themselves, by a government that just told them no. Nobody in this story is easing off the gas. What Amodei got in one weekend is a passenger in the front seat, allowed to read the speedometer and say the number out loud. It's less than the title promised, and already more than Washington extracted by force in June.