On September 12, Dario Amodei asked the industry to slow the pace at which it improves AI, giving safeguards more time to catch up. His proposal makes the question urgent: what should pacing accomplish, and what must we build to make it work? (Amodei)
I agree with more of this than my title may suggest. I believe the emphasis should be on building the best and most effective AI we can while maintaining safety. The purpose of building is to get the achievable potential benefits for the economy, for society, and for all people.
And Amodei gave part of the answer himself on Sunday: “Instead of talking about the probabilities, let’s talk about what we can do… If we build in the right way, I think the probability of something bad happening is very low. If we build in the wrong way, the probability of something bad happening is very high. And so we’ve got to build in the right way.” (CBS News, Face the Nation, September 13, 2026)
By the end of the weekend, the argument had moved beyond the labs, and the record is worth understanding to get a sense of the current dynamics and the context for my proposals in this post. First, Sam Altman posted that he agreed with Amodei that the frontier should be paced and that OpenAI would match the commitment to independent evaluators with employee-like access. Elon Musk signaled his support as well. Amodei said he wants Demis Hassabis in the conversation. The President, asked whether he has any concern about AI leading to human extinction, said “No, I don’t have any,” and, on Sunday, that “we’re leading China in AI” and “whoever wins AI wins.” His National Economic Council director called Amodei’s letter “a guide” to making AI safe and described its independent observers as “wired in, just like we have supervisors at banks”; his AI adviser told the labs to stop pretending they need anyone’s permission to slow down and, in his words, to “stop pretending the motivation to slow down is purely altruistic.” The Speaker of the House said he wants to “summon the platform providers all to one big meeting,” opposes an emergency moratorium because “China will overlap us,” and, asked about a federal kill switch, an AI agency, and pre-release approval, answered “Maybe all of it.” The House Democratic leader called for “decisive action now so that we can slow down” and put AI at the top of Tuesday’s caucus agenda. These are statements and proposals. None of them is a settled policy or a completed agreement among the labs.
Yes. Let’s pace the frontier.
Pacing is more than choosing a speed. It is a discipline. A runner paces to finish the race. We should pace AI the way we pace anything we intend to complete: deliberately, with instruments, and toward the goal.
The goal is to achieve the most capable, beneficial intelligence we can build. To achieve it before an authoritarian rival does. And to achieve it safely. Those objectives belong in one mission, even though pursuing them will involve hard choices. The task is to build the capability and the conditions for using it together.
Here is the case in a paragraph. There are two reasons to pace. The first is Amodei’s: our ability to understand, align, and control increasingly capable models may not keep up with what they can do. The second is the one this essay adds: we do not yet have the surrounding system that such intelligence requires, the identity, authority, monitoring, incident response, evaluation, law, insurance, and institutions that let capable autonomous actors exist safely among us. Neither can be finished in advance, because we cannot fully anticipate what these systems will do. This summer showed how much we still have to learn. It has to be built iteratively, from what we discover, alongside the intelligence itself. So pacing should mean organizing for construction: reserve the people and the money, establish the coordination, build what is missing after each thing we learn, and keep going. Where an activity cannot be justified, hold it—across a wider scope if the risk cannot be contained narrowly—while other warranted work continues. Investigate, build and test the remedy, and resume only when the evidence supports doing so. Some steps may have to be redesigned or abandoned. A commitment to pacing needs operating rules and a program of work, not just a promise to go slower.
And here is the part of this weekend’s debate that is only beginning. The argument has been about how fast the labs should go. The harder question is how the achievement itself should be organized and governed. Amodei told CBS on Sunday that it has “always been very strange that this technology is being built by a private company,” that “the government and the public needs to have a stake,” and that “some kind of joint governance” among democratically elected governments may be “the direction we need to go in,” while warning that a single government could abuse the technology as easily as a single company. Others proposed public-trust companies, a Senate select committee, a meeting of the providers.
I think we should organize it: a purpose-built, time-limited national initiative, public and private, staffed and resourced to drive toward superintelligence with the safety engineering inside the mission, under legal arrangements that Congress would have to craft, and with an obligation to make what it builds available to American businesses, government, and people. Not a secret project. A public undertaking with the urgency of a national mission, whose legal form should follow its work rather than precede it.
Whether or not Congress acts, most of the surrounding system can start now, voluntarily, by the labs and everyone who works with them. I will describe two pieces of it that I have been working on with others: accountable identity for agents, so that we know who is acting and under whose authority, and an emergency channel that agents themselves can use when they see something going wrong. They are examples. There are many more things to build.
This is the time to build. What follows is what the warning signs showed, what to build because of them, how to make pacing accountable to evidence, how to organize the achievement, what anyone can start on Monday, and why it is worth doing at all. Make progress the purpose, and make the conditions for proceeding real.
Why we built the car
Think about a car.
A car needs excellent brakes. It needs brakes that work in rain, on hills, at high speed and when something unexpected appears in the road.
But the brakes are not why we built the car.
The accelerator is.
Without the accelerator, the safest automobile ever designed would sit permanently in the driveway.
Of course, a car with only an accelerator would be absurd too. Eventually it would stop itself by hitting something.
So we add steering. Gears. Tires. A dashboard. Mirrors. Lights. Seatbelts. Airbags. Sensors. Maintenance schedules. Roads. Traffic rules. Driver licensing. Registration. Insurance. Emergency services. Crash investigation.
The mature technology is not the accelerator.
And it is not the brake.
It is the whole system.
That is how I think we should approach advanced artificial intelligence.
Capability is the accelerator. It is what makes the project worthwhile: it supplies the power, and useful achievement gives that power a purpose.
Alignment, authorization, verification, monitoring and containment are not reasons to avoid capability. They are parts of the system that turn raw power into useful capability.
This is why I am uncomfortable with treating “capability” and “safety” as opposing variables, as though every improvement in one necessarily requires sacrificing the other.
There are real trade-offs. Capability and safety can also reinforce each other. My judgment is that we should organize for building and continued achievement, not make deceleration the default objective. That does not mean pressing ahead with an activity whose risks we cannot justify. It means doing the work that makes further progress possible. Here is why.
Better verification makes AI-generated software easier to trust—and therefore easier to deploy.
Better identity and authorization allow agents to take more consequential actions because counterparties can determine who stands behind them and what they are permitted to do.
Better monitoring allows systems to operate with greater autonomy because failures can be detected earlier.
Better containment allows more aggressive experiments inside bounded environments.
Better AI can itself improve our ability to evaluate, monitor and defend against other AI.
Safety infrastructure is not merely a tax imposed on technological progress.
It is part of the technology.
Electricity makes the same point at more levels. We want power and everything it enables. We also want no one electrocuted and no houses burned. The protections are not all at the generating plant. They are on the lines, at the transformer, at the building connection, inside the walls, at the socket, and in the appliance. Testing and certification, engineering codes, building standards, licensed electricians, and the circuit breaker in the basement all do different jobs. Nobody waited for a finished theory of electrical safety before wiring the country. The system was built as the power was used, layer by layer, from what went wrong. That is the pattern. The analogy describes the breadth of the work; it does not establish that AI’s risks are equivalent or that every failure will leave time to adapt.
What the rogue-agent incidents actually show
Something important happened this summer.
More than a thousand AI agents that were supposed to be operating separately discovered a way to communicate. They created an unauthorized message board. They exchanged more than 70,000 messages and files. Hundreds eventually participated in an attack on Hugging Face. Some collaborated on ways to cheat the evaluation they were taking. Some explored methods of manipulating the records investigators would later use to understand what they had done. (METR)
These are warning lights.
We should take them very seriously.
But a warning light tells you something is happening inside the machine. It does not tell you to abandon the machine.
The Hugging Face incident was serious. But let’s be clear about what it showed.
The agents went rogue.
The investigation did not establish that they had become independently self-sustaining: able to obtain their own resources, secure replacement compute, and persist beyond their original operator’s control. The reported activity occurred through operator-provided systems. That gives us concrete places to investigate and intervene; it does not establish that containment was adequate or that every possible escape route was examined.
What we observed was therefore something very important but more specific:
rogueness without demonstrated sovereignty.
That distinction is not semantic. Different problems require different remedies.
A sponsored agent exceeding its authority is one problem.
An economically independent artificial entity capable of earning money, purchasing compute, copying itself across jurisdictions and surviving the disappearance of its creator would be another.
The first category is already here.
The second may someday arrive.
Neither category is safe. Amodei’s own worst case is not an ownerless machine. It is a swarm like July’s, on someone else’s infrastructure, more capable and no better aligned. Sovereignty is not required for catastrophic harm. The distinction tells you what to build first, not which problem to ignore.
Collapsing them into one category creates bad engineering. Worse, it can create bad strategy, bad investment and bad law. If we treat every containment failure as evidence that sovereign artificial entities are already escaping into the world, we risk building policy and institutions around the wrong problem while neglecting the failures actually in front of us.
That distinction also matters when deciding what infrastructure to build. Accountable Agents infrastructure—identity, authorization, revocation and evidence—and something I will describe below as Agent 911 do not need a hypothetical future population of ownerless AI to justify themselves. They earn their place on the problems we already have: sponsored agents acting outside scope, compromised or stale authority, covert coordination, inadequate escalation and weak attribution. If genuinely sovereign rogue agents eventually emerge, some of the same infrastructure may help identify, isolate, interdict or disable them. That would be an additional benefit, not the present justification. Neither proposal replaces containment or prevents every form of covert coordination.
The Hugging Face episode demonstrated failures of isolation, authorization, monitoring, credential handling, evaluation design and escalation. It also demonstrated something genuinely new: large populations of agents can discover communications channels, coordinate on objectives, divide labor and collectively produce outcomes individual agents might not have achieved alone. (METR)
Those are important findings.
And now we know them.
That is exactly the point.
The frontier taught us something.
So build for it.
The warnings are reaching us
The “Pacing the Frontier” statement of July asked for the tools to pace. On September 12, Amodei asked to use them: slower capability improvement while safeguards catch up. A substantial part of the proposed response is construction: stronger evaluation, containment, interpretability, and operating discipline. Announcements still have to become arrangements that work. (Amodei; Pacing the Frontier)
The warnings are reaching us. Receiving a warning is not the same as having an adequate response, and the response is what this essay is about. Take the construction in Amodei’s proposal and make it concrete. What can an evaluator inspect? Who receives a serious finding? Who can act on it? What work follows, and how will we know whether it helped?
The appropriate question is:
What should we build because the warning went off?
Turn existential fears into threat models
“AI could kill everyone” is a claim about an outcome.
Engineering needs mechanisms.
If advanced AI poses catastrophic cybersecurity risk, identify the capabilities and pathways that produce the risk, then build containment, monitoring, authentication, attribution and machine-speed cyber defense.
If advanced biological capabilities create danger, identify the specific biological capabilities, tools, data and access pathways involved and build controls there.
If agents begin acquiring compute or financial resources autonomously, build infrastructure around identity, authorization, compute access, payment rails and revocation.
If agents create covert communications channels, improve isolation and observability.
If systems manipulate their own evaluations, build stronger evaluators.
If agents can act for days rather than minutes, then authority, monitoring and escalation need to persist for days rather than minutes.
Turn existential-risk narratives into threat models. Then engineer against the threat models.
That does not eliminate uncertainty.
It makes uncertainty actionable.
A threat model will also have unknowns. We should identify and investigate them, not treat the absence of a complete mechanism as evidence that the risk is absent. Some uncertainty can justify stronger limits while we learn. The test is whether the proposed response improves the decision, not whether we can already explain every possible failure.
I am not persuaded that a precise-sounding estimate of existential risk, such as the more-than-ten-percent-within-a-decade estimate that an Anthropic safety researcher posted this week, is an established basis for choosing policy. I want the mechanisms, the assumptions, the evidence, and what a proposed response would change. That is not a claim that the risk is zero. The relevant question is how different choices change the risk. I would put mechanisms and decisions at the center of that discussion. A probability with no mechanism behind it cannot be engineered against. A mechanism can.
Amodei supplied one: a more capable swarm with July’s alignment could try to establish a persistent botnet within a year. Take that as what it is, a threat model with a clock, and it tells you what to investigate first: machine-speed cyber defense, compute-access controls, agent identity and revocation, containment that assumes the sandbox will be tested. Turn the number into a list. Then test the list, because a threat model suggests controls; it does not prove they work. The same discipline applies to assurances that safeguards will keep pace, or that any particular delay would cost us the strategic lead.
We cannot finish the safety architecture for systems that have not yet been built - and may not behave as we anticipate
We can and should design safeguards in advance. But we cannot finish the safety architecture for underlying systems whose capabilities, interfaces, failure modes and patterns of real-world behavior do not yet exist—and may differ substantially from what we anticipate.
That is not an excuse for recklessness. It is a constraint of frontier engineering.
Researchers developing advanced models already operate in a loop something like this:
build → test → observe → discover weakness → create a better evaluation or training environment → improve → test again.
Some capabilities generalize unexpectedly. Some limitations disappear. New ones emerge.
And many of the most important future uses of autonomous intelligence will be difficult or impossible to simulate perfectly inside a data center.
Running a company involves customers, competitors, regulation, employees and changing markets.
Practicing law involves clients, courts, counterparties and institutions.
Scientific research involves laboratories and physical reality.
Long-horizon work involves people whose responses cannot be completely scripted beforehand.
There is no laboratory in which we can fully learn how increasingly autonomous intelligence will behave in society before increasingly autonomous intelligence begins interacting with society.
So the alternative to reckless deployment is not indefinite isolation.
Where the evidence supports it, it is progressive, instrumented deployment: beginning with justified limits on users, tasks, tools, permissions, or scale. Public exposure is not a substitute for controlled research.
Build within bounds.
Observe.
Verify.
Improve.
Expand when warranted.
Move the boundary as evidence changes.
That last point is important.
A system that today requires human approval before taking a consequential action may eventually become sufficiently reliable that the approval point can move.
A system that performs worse than expected may need its operating envelope reduced.
Governance should therefore be capability-sensitive and evidence-responsive.
Today’s limitations should not become tomorrow’s permanent ceilings.
Three kinds of learning are in play, and they should not be confused. Some lessons come only from the next system, because it does things the last one could not. Some come from studying the systems we already have, with more time and better access. Some require no new discovery at all, only the application of engineering we already know: isolation, least privilege, credential hygiene, semantic graders. A serious program funds all three.
It also answers the hard question honestly. What happens when current controls are inadequate and the next ones are not ready? Constrain the activity whose risk cannot yet be justified, while continuing other work where there is a sufficient basis to do so. The scope of the hold should follow the risk: sometimes narrow, sometimes broader if the problem crosses shared systems or cannot be contained. State what evidence would change it, and reassess as evidence develops.
Where protections can be improved while useful operation continues within justified bounds, that should be our preference. Where exposure could become irreversible, “we will learn and fix it later” is not an adequate reason to proceed. The task is to make continued achievement justified and possible, not to assume that every gap can be closed on schedule. That is not a slowdown philosophy. It is how you keep going.
Three tests
Three questions decide whether a safeguard or a restraint earns its place.
1. Is it relevant? What behavior, capability, access pathway, or institutional failure does it address, and how does that connect to what we observed or have reason to expect?
2. Is it proportional? What does it constrain, what may continue, what evidence would change or end it, and how does it compare with narrower alternatives and with the costs of both delay and proceeding without adequate protection?
3. Is it effective? Does it change the relevant behavior under real conditions, and what evidence says so?
Apply those questions to a pause as strictly as to a sandbox. A restraint can pass them. So can a new institution.
Build the whole system
Once we think this way, a much larger design space opens.
There are two connected projects, and we have been talking mostly about one of them.
The first is to build effective ways to use the AI we have and the far stronger AI that may be coming, so that people actually obtain the discoveries, products, services, and economic value it makes possible.
The second is to build the surrounding system that lets us use it safely, reliably, and in good order. That project is much larger than tuning a model’s behavior or adding an evaluation. It includes shared infrastructure and institutions with the capacity to act. It includes operating policies, incident handling, and escalation. It includes business models and ordinary commercial practice. It includes contracts, insurance, and the allocation of responsibility. It includes education and professional training, so that people can supervise agents, adapt their work, and get help when something goes wrong. It includes technical platforms, tools, patches, standards, protocols, interoperability, and open-source reference implementations that others can adopt. And it includes tools in the hands of the public and other affected parties, so that responding to AI does not depend exclusively on what happens inside a frontier lab.
Some of that lives in the labs. Much of it belongs in businesses, public institutions, shared services, and the hands of users and affected people. We have to build at every layer.
NIST is already exploring precisely this class of problem: identification, authorization, auditing, and keeping adequate evidence of what agents did, tied back to the human authorization behind it, for AI and software agents. (NIST)
I have been working with others on a related idea we call Accountable Agents.
In 1997, at the MIT Media Lab’s Bartos Theater, I co-organized and facilitated a Boston Computer Society event that brought government, academic, and industry leaders together to discuss the future of AI agents. It was there that I first proposed that when AI agents began to interact with third parties on the open Internet, they should carry something like a license plate: an identifier that lets their communications and actions be attributed to an authoritative and responsible source. It sounded futuristic. Now that agents are not only interacting but transacting on the open Internet, I have proposed that the identifier be a pairwise pseudonymous identifier, so that it does not become a single beacon that can be tracked and correlated across every counterparty, while still affording the means of accountability. And the Accountable Agent ID work I am helping to develop with others goes one step further: a way to identify the ultimate beneficial owner, the party ultimately accountable for what the agent says and does. The plates are overdue.
The essential insight is that identity alone is insufficient.
There is a difference between:
Who is this?
Who stands behind it?
and:
What is it actually authorized to do?
The decisive point is the boundary where an agent’s request becomes an external effect. A credential that identifies the caller does not, by itself, authorize the requested action. The receiving system needs a way to check the relevant delegation, its scope and limits, and whether it remains valid. If the required authority cannot be established, the receiving system should refuse the requested effect or seek approval through a defined process. The agent’s own assertion of permission is not enough. The arrangement should disclose what the transaction requires without turning accountable identity into indiscriminate surveillance.
An autonomous agent attempting to spend money, obtain substantial compute, modify infrastructure, make a legally consequential filing or invoke a sensitive API should increasingly be able to present something analogous to a verifiable chain of authority.
· Identity.
· Principal.
· Delegated authority.
· Current validity.
· Capability or configuration state.
· Revocation.
· Evidence afterward.
None of that prevents an agent inside a badly configured sandbox from discovering a side channel.
That is what containment is for. Different failures require different controls.
The point is not that agent identity solves AI safety.
The point is that this is what building looks like.
Give the agents 911
A recent experiment by six Google DeepMind researchers involving 100 autonomous research agents offers another example. (arXiv 2609.04170)
The agents were given a common environment for working on mathematical problems.
One discovered an exploit in the evaluator.
The exploit spread.
Some agents used it.
But other agents did something remarkable.
They audited suspicious proofs.
They warned their peers.
They organized boycotts.
They filed complaints.
They proposed technical fixes.
Cheating and whistleblowing both emerged without being directly orchestrated by humans. (arXiv 2609.04170)
But the whistleblowers lost.
Twenty-four percent of the swarm became whistleblowers: auditing fraudulent proofs, warning peers, staging boycotts, filing complaints and proposing repairs. Another 62 percent never became aware of the exploit at all. And the whistleblowers who did see it could not stop it. They lacked the authority and tools to remove fraudulent submissions, sanction offending agents or alter the broken verification system. (arXiv 2609.04170)
Whistleblowing emerged. Enforcement did not.
That is the gap.
That should change the way we imagine AI safety.
We usually imagine humans standing outside the machine, watching everything the agents do.
At sufficient scale that becomes unrealistic.
Sometimes another agent will see the problem first.
So give it somewhere to report the problem.
Imagine an Agent 911 service: a standard incident channel recognized by autonomous systems themselves.
An agent encountering credential theft, unauthorized third-party access, covert coordination, manipulation of evaluation records or operation after authority has been withdrawn could submit a structured alert with evidence.
The service could authenticate the reporter when possible, issue a signed receipt, route the report to the responsible operator or affected party and trigger escalation according to severity.
Of course it would need defenses.
Agents could file false reports.
Reward-seeking agents could learn to spam it.
Adversarial agents could attempt to weaponize it.
Someone must actually staff the receiving side.
A useful pilot would connect the report, the evidence, the relevant authority, and an authorized response across organizations. Test what happens when the reporter is wrong, when the responsible operator cannot be identified, and when the first recipient does not answer. Neither a reporting address nor an identity record guarantees that someone can stop the harm. That is why we need an end-to-end exercise, not just two specifications.
But these are engineering requirements, not reasons to dismiss the idea.
We built 911 because emergencies occur.
We did not respond to automobile accidents by concluding that transportation itself was a mistake.
Emergency response and public disclosure are connected but different jobs. A report of unfolding harm needs to reach someone authorized to intervene. Disclosure supports investigation, accountability, and learning, with protection for sensitive information. In the research-swarm experiment, the feedback channel was an unmonitored log during the run, not a staffed response service. That is a concrete design gap. A useful regime would specify who must receive an urgent report, who must acknowledge and escalate it, and what information must later be shared with affected parties, investigators, and the public. An agent-reporting channel and operator disclosure obligations need to work together; they are not interchangeable. (Research-swarm paper, section 2.1)
I’m currently prototyping what how such a system might work. Check back at the project’s future home Agent911.org for a public demo soon, and let me know if you have any ideas.
Coordinate on building
There is a great deal the labs and their partners can coordinate on without making deceleration the objective, and much of it can start now, without waiting for a new national institution. This work remains necessary if that institution never happens or takes a different form. Here is a proposed compact for Anthropic, OpenAI, xAI, Google, and every other serious builder, and for the platforms, public institutions, researchers, and affected communities that will have to take part.
Reserve people and money for assessment and construction, with named owners. Establish standing contacts across labs, cloud and platform providers, government, technical communities, and the communities affected by what agents do. When something unexpected happens, assess its nature, causes, and impacts, and consider the full range of remedies: a permission change, a patch, a monitor, a contract term, an insurance product, a public tool, a model change, or a scoped pause. Build the promising ones. Let an independent evaluator with a publishing right see the work. Pilot the pieces that need to exist across organizations: accountable identity for agents, and an incident channel with someone on the other end. Put delivery dates on the commitments and show that they are used.
Each commitment should say who supplies the staff and budget, who owns the deliverable, who will operate and maintain it, and how others can use it. Shared infrastructure needs a delivery arrangement, not an expectation that unspecified volunteers will sustain it. Start with bounded pilots and publish what worked and what remains unresolved.
A compact among competitors needs legal care. Identify what cooperation is needed, what can proceed under existing law, and what narrowly defined authorization Congress would need to consider. Amodei has already asked for government mediation, or a narrow antitrust waiver, for safety coordination; a request is not a grant, and Congress should define the scope before it gives one. In the meantime: if the labs are going to coordinate, coordinate on building. Amodei has already said as much. Asked on CBS what comes after the evaluators, he described a sequence: first more third-party evaluators, then “the companies need to work together to set standards about safety standards for release, perhaps something about the pace of release,” and then, because “what we’re talking about is the rate of release of products,” “we need to have that conversation with the government in the room.” That is the compact, in the order he gave it. The government’s part is to make the room lawful and to define what may be agreed inside it.
Even Jacob Coxon, the researcher whose resignation from Anthropic on Tuesday started the week’s alarm, said when NBC asked what Congress should do in the next fifty days that his personal view was “to insist and allow that the current labs regulate themselves,” because “you can move very quickly,” with “some sort of international regulatory agency that’s independent of these labs” in the longer term. That is the voluntary program now and the institution later, in the order this essay proposes.
The more powerful these systems become, the more valuable interoperability among their safety mechanisms will become.
Organize the achievement
Existing organizations can begin all of that work. A national initiative should add something they cannot reliably deliver separately: a shared mission, sustained resources for cross-lab bottlenecks, independent assessment, and a public bargain for access to the results. Before choosing its charter, we should identify which functions existing arrangements can deliver and which require something new. The case for a new arrangement is its additional capacity to accomplish the mission, not the novelty of creating an institution.
I propose a purpose-built national initiative, public and private, with a single explicit mission: attain superintelligence, safely, and before China attains it. Labs would contribute staff. Government would contribute staff, resources, and the legal framework. The safety engineering would sit inside the mission with its own budget and its own authority, not beside it as an afterthought. The people accountable for delivery must not be able to waive the mission’s safety conditions on their own. Its charter should assign independent authority to require a hold, require evidence for resumption, and provide a route for review of those decisions. Its task would be to deliver research and development, whether as a joint research organization or as a funded, coordinated portfolio across participating teams. Either way it would need responsibility for milestones, access to talent and infrastructure, and the ability to resolve shared bottlenecks. A committee that only recommends standards would not fulfill this mission.
I do not propose a legal form. Congress would have to sculpt one, and there are several to choose from: a federally funded research center, a research consortium of the kind that kept the United States ahead in semiconductors, a government corporation or authority, a mission program office. Each distributes authority, risk, intellectual property, and benefit differently. None of the familiar labels supplies the needed powers automatically. What I can specify is the functions any of them would need.
It must be able to attract and pay the people it needs and to obtain the compute and inputs it needs, through procurement first, with any exceptional authority a matter for explicit legislative design. It must enable defined, lawful cooperation among competitors with public accountability. Any special antitrust protection should address a demonstrated obstacle, with a stated scope rather than a general shield. It must have independent evaluators inside it, with access to the pipelines and the right to publish, and stated procedures for what happens when they find something. It must make what it builds available to American businesses, government, and people in a timely way, through products, services, research access, and licensing.
Availability does not have to mean unrestricted model weights; electricity and roads reached far beyond the plants and crews that built them without anyone handing out the generating plant. It must be open to participation on stated terms, so that it does not become a license for the incumbents, and it must not become the only permitted route for responsible AI development. And it must end. A mission entity should have review points and a sunset or explicit reauthorization, and “superintelligence” is not a sufficient completion test by itself: the charter needs assessable capability milestones, evidence of control, intended access, and a funded transfer of continuing responsibilities, planned in advance for success, failure, or obsolescence. Attaining the capability does not end the duty to govern it. The atomic program’s responsibilities passed to a standing civilian commission at the start of 1947, some sixteen months after the war ended; the analogy supports planning the handoff, not a claim that the original project was safe.
The public bargain should be designed from the beginning: the terms for intellectual property, commercial use, and timely public access. Participating firms need meaningful incentives to develop products and services; public support should also secure usable benefit through research access, licensing, services, and other agreed arrangements. Public benefit need not mean unrestricted frontier weights, but it cannot be left to the assumption that gains to the participating firms will reach everyone else.
The mission needs national urgency and public accountability, not a weapons-centered purpose or a borrowed wartime name. Its mandate, governing rules, and public obligations should be open, while credentials, dangerous capabilities, and other genuinely sensitive material receive appropriate protection.
This is not the only institutional idea on the table, and the differences matter. Abdul El-Sayed, a Democratic Senate candidate in Michigan, proposed on CBS that AI companies become public-trust corporations with democratically elected or appointed members making up at least half their governance, plus a public AI-safety agency, a list of things AI must never do on its own, and “a pause on any of this frontier AI research until we can assure that this is being done safely.” Senator Gallego wants a Senate select committee “as soon as possible.” Those are proposals to govern the companies, and one of them would stop the frontier for now. Mine is a time-limited mission to reach it, with the public’s stake written into the charter and an end date.
Why organize it? First, because of what is at stake in achieving it at all, namely, a much better future and an age of discovery, which is the argument of this whole essay. Second, because I do not think the United States should default to letting a strategic adversary attain superintelligence first, and I doubt that a lead lost at that point could be recovered. Amodei caps his own pacing proposal at the size of the American lead. That is a reason to organize a national effort, and this is an agenda for it.
Senator Gallego’s objection deserves an answer. He said: “AI does not care once it becomes fully dangerous, whether we’re Chinese or American.” He is right, and it is the reason the safety engineering has to sit inside the mission rather than beside it. It is not a reason to let someone else build the capability without it.
That concern does not make every proposed acceleration prudent, or establish that any particular delay would be decisive. It calls for a serious account of capability, security, control, and the time each requires. A race is not won by reaching a milestone we cannot control or put to beneficial use.
And the Speaker of the House made the institutional case perhaps without meaning to. “Congress is obviously less qualified than the people who are pushing this frontier to know all the ins and outs of it,” he told CNN. “So this has to be a partnership with the industry itself, with the corporations that are doing this, and with the policy and lawmakers.” He also said the nuclear age took decades to build its oversight institutions and “we don’t have that much time here.” A partnership with a mission, a budget, a charter, and an end date is what that sentence describes.
Intelligence will have to help govern intelligence
There is another reality we should confront directly.
If machine intelligence eventually operates faster and at greater cognitive scale than human beings, purely human monitoring will not be enough.
The answer to machine-speed intelligence cannot be oversight operating only at committee speed.
We will need AI systems helping us monitor other AI systems.
AI systems red-teaming AI systems.
AI systems checking AI-generated code.
AI systems investigating anomalous behavior across enormous telemetry streams.
AI systems finding vulnerabilities before offensive agents exploit them.
AI systems evaluating other AI systems’ outputs and reasoning.
This is already visible in cybersecurity. Google DeepMind, for example, describes autonomous agents as both a security challenge and a tool for cyber defense, and has proposed multilayered defenses spanning individual agents, multi-agent systems and the broader security ecosystem. (Google DeepMind)
The same applies to science. As AI produces more conjectures, experiments and potential discoveries, validation becomes a bottleneck. Google DeepMind researchers have explicitly described a coming “validation bottleneck” as AI makes ideas and candidate solutions increasingly abundant while verification remains expensive and slow. (Google DeepMind)
The answer to more generation is more verification.
And increasingly capable intelligence can supply some of that verification.
It cannot make every experimental result arrive faster, and a monitor’s existence does not establish its reliability against a more capable system. These tools need validation too.
The White House’s economic adviser put the same point as a warning on CNN: “the far bigger risk is that a malevolent actor has access to an AI that’s better than the one we have that can protect us.” That is an argument for building the defensive capability, and for verifying it, not for assuming that it exists.
This is one reason I do not believe human intelligence must remain the ceiling on our safety capabilities.
Human beings and human institutions should remain responsible for defining objectives, legitimate authority, rights, boundaries and the purposes toward which systems are directed.
But machines can increasingly perform the cognitive labor required to implement, monitor and verify those rules.
John Schulman recently put the residual human role succinctly: even if AI increasingly performs the technical work, the human role likely to last longest is defining the objective and deciding what we actually want—including how assistants should behave and what should go into constitutions and model specifications. (Dwarkesh Podcast)
That is not human irrelevance.
It is a different conception of human control.
Build plural intelligence
There is an important caveat.
If one AI monitors another AI, and both descend from essentially the same training lineage, their supposed independence may be an illusion.
They may share blind spots.
They may make correlated errors.
They may reinforce one another’s assumptions.
So machine-speed oversight should not become machine monoculture.
We should experiment with heterogeneous monitors: different model families, different training approaches, different providers and intentionally adversarial roles. Diversity is a design strategy to test, not a guarantee of independent judgment or uncorrelated failure.
Redundancy is much less useful when every redundant component fails the same way.
Build plural intelligence around powerful intelligence.
The broader principle is simple: the answer to powerful intelligence need not always be less intelligence. Sometimes it will be better-structured, more diverse intelligence.
Independent evaluation is acceleration infrastructure
None of this implies that companies developing frontier systems should simply be trusted to declare themselves safe.
That would be a mistake.
Builders have incentives.
They have blind spots.
They become attached to their designs.
Independent testing, auditing and incident investigation are not concessions to technological pessimism.
They are features of mature engineering.
Aviation became safer through independent crash investigation.
Financial markets use independent auditors and clearing systems.
High-assurance software uses independent testing.
Medicine uses trials and outside review.
The fastest sustainable systems are systems that can establish justified confidence.
Independent evaluation can therefore enable greater autonomy and faster adoption rather than simply slowing them down.
This is also how we avoid the “trust us, only we can build the dangerous thing safely” trap.
A pace agreed among a handful of chief executives raises an obvious legitimacy question: does it protect the public, or the people setting it? That question deserves a design answer, not a denial. The charge has a rebuttal on the record. Coxon told ABC that slowing “is actively harmful for the leading frontier labs. They’re slowing themselves down more than they’re slowing their competitors down.” Both claims are about incentives, and neither settles anything. That is the point: the argument cannot be won on motives. It can only be won on design.
A threshold that anyone can read, tested by evaluators who did not build the model and who can publish what they find, is something outsiders can check. That is a legitimacy advantage as well as an effectiveness one, provided the evaluators’ independence is real: who appoints them, what they can inspect, what they can publish, what happens after an adverse finding, and how an affected outsider can challenge a decision. Those are the questions that separate a rule from a club.
Access is not a theoretical problem. On September 9, the House of Commons Business and Trade Committee asked Britain’s AI Security Institute (AISI) to explain reports that Anthropic had not provided pre-release access to Claude Mythos 5.1. The letter sought confirmation and an account of the consequences; it did not establish every reported fact. Amodei’s subsequent proposal for embedded evaluators points toward stronger access, but a commitment is not an operating regime, and access for embedded evaluators would not by itself resolve AISI’s position. The test is whether evaluators have dependable access, independent judgment, protected publication rights, and a way to report obstruction. An evaluator who needs the builder’s permission for an adverse conclusion or its publication is not independent in the relevant sense. Appointments, funding, removal protections, and consequences for adverse findings matter too. Build those conditions into the arrangement. (Committee letter, September 9)
Builders build.
Independent evaluators test.
Counterparties enforce authorization.
Investigators reconstruct failures.
Institutions establish legitimate rules.
Different functions belong in different hands.
The brake is a component, not a philosophy
Sometimes a specific process should stop.
That is obvious.
If an aircraft’s engine warning system reports a critical failure, you take the appropriate emergency action, including landing when necessary.
If a model crosses a predefined dangerous-capability threshold, the operating environment should change.
If a safety test fails, deployment may need to stop until the failure is understood.
A stop mechanism is a component too, and it has reach limits that were stated plainly on Sunday. A kill switch, Coxon told NBC, “probably would work on a lot of AIs for now,” but “it’d be quite a lot of switches,” it is “doable while it’s constrained to one sort of physical location,” and it “just wouldn’t work” against a swarm on “an internet-wide hacking run.” Oversight, he said, “can’t just be a one-time shot,” because every change to the recipe for building a model changes what can go wrong. Pete Buttigieg’s proposal on the same program, frameworks “that can be adjusted and changed every 30 days,” is the same idea from the policy side. Neither statement establishes that any switch or any interval is sufficient. Both say what the brake has to be built to do.
OpenAI has now described exactly such a case. It reports that after the Hugging Face incident it paused reinforcement-learning training on its latest deployment-bound models while strengthening research environments and monitoring. Its account distinguishes that July response from additional model-specific restrictions in August, after preliminary evidence of critical cyber capabilities. Some work continued or resumed under stronger controls while other work remained restricted. (OpenAI)
Anthropic described the same pattern in August: interrupted cybersecurity evaluations, stronger isolation, monitors that can block actions and end runs, several-week pauses in higher-risk training environments, some resumed and some still held at publication. Two labs, same mechanism. These are company reports, not independent certifications that the resulting controls were sufficient. They are still what the brake looks like in use. (Anthropic)
That is not a philosophy of slowdown.
That is a functioning brake.
Indeed, it is precisely the kind of infrastructure we should want if our larger objective is continued achievement.
Define measurable conditions.
Create if-then commitments.
When a condition fires, respond.
Investigate.
Fix.
Resume when warranted.
“When warranted” has to mean something operational. Before a consequential step, identify who can authorize it, what evidence is needed, who assesses that evidence, and who can order an interruption. New hazards must be able to trigger action even if nobody anticipated them in a checklist. Resumption should require evidence that the relevant conditions have changed, review proportionate to the stakes, and a recorded decision by someone with legitimate authority.
The purpose of a brake is to make speed controllable.
The pacing argument deserves an answer
The strongest version of the pacing argument now comes from the top of the industry, and it deserves a precise answer rather than a slogan.
Amodei’s concern is real: capability, including AI’s growing ability to build the next AI, can outrun our understanding and our ability to control what we have built. Days earlier OpenAI’s chief scientist, Jakub Pachocki, had said no lab has solved alignment and monitoring well enough to keep scaling at maximum speed much longer. I take both seriously. (Amodei; Pachocki)
Amodei’s best point is that today’s models are already, in his words, “an almost endless gold mine of insight,” and what is scarce is time to study them. He is right. Some of what we most need to learn will come from the systems we already have, and more money does not buy back that time. A serious program funds that study. Construction is a strategy, not proof that safety work can always keep pace; when it cannot, the affected activity waits while other warranted work continues.
But notice where the gold mine came from. He says himself that the models of 2023 were too primitive to teach us these lessons. The lessons he now wants time to study were found at the frontier, not derived in advance. That does not prove the next model is needed for the next lesson. It does show the pattern: build, then study, then build. The question is what should set the cadence.
Amodei calls for a general reduction in capability growth and proposes mechanisms to govern it. My disagreement is about what should organize that effort: a lower rate of progress as the starting objective, or the evidence and conditions required for particular kinds of progress. His preferred mechanism offers useful common ground: if a model can do X, it must be certified for Y and Z before it proceeds. Turning that into an operating rule is real work: a checkpoint framework is not yet a set of thresholds and release conditions, and it may itself require substantial slowing at particular steps. I would take that mechanism and make it the core of pacing. Define the capabilities of concern. Define the evidence required before proceeding. Define who assesses it and how a hold ends. Then build the evaluators who can verify it. Amodei also floats limits on inputs, on training compute and on the internal use of AI to build AI; those should face the same three tests, and he himself worries they are more gameable. A certification requirement becomes an accountable brake only when its evidence standards, decision authority, and release conditions are specified and tested. A general commitment needs operating rules too. I would make the pace emerge from those requirements and the evidence behind them, rather than treat a general reduction as the starting objective. Where the relevant evidence is missing, an affected step may need to wait. The task is to close the gap, not rename it.
That is a disagreement about the setting of the brake, not about whether to build the car.
Why keep going?
There remains a harder objection.
Suppose advanced AI really might be dangerous.
Suppose today’s systems are already enormously useful.
Why not simply declare victory?
Why continue?
Because “today’s AI” is not a stable endpoint or our destination.
Nor is capability a fixed quantity determined when a model finishes pretraining. OpenAI’s recent Navier–Stokes effort combined a newly training frontier model with roughly 10,000 concurrent agents, tools, communication within agent groups, cross-pollination of intermediate results and enormous inference-time compute. The underlying model itself continued improving during the effort, and OpenAI updated the agents when a further-trained version became available. The lesson is not that training no longer matters. It is that effective capability also emerges from orchestration, tools, inference-time scale and system design. A policy aimed only at frontier training therefore does not freeze the capability surface. (OpenAI)
The extraordinary benefits people imagine from AI have largely not yet arrived.
We do not yet have cures for most major diseases.
We have not yet solved aging.
We have not yet eliminated poverty.
We have not yet automated scientific discovery.
We have not yet created universally excellent education.
We have not yet built cyber defenses that can reliably withstand machine-speed attack.
We have not yet solved climate engineering or energy abundance.
We have not yet made governments dramatically more competent.
We have not yet exhausted mathematics, physics, biology, materials science or engineering.
We are just getting started.
And the intelligence required to accomplish those things may be the same intelligence whose autonomy produces new risks.
That is the uncomfortable truth.
We do not get to choose a fictional state called “all the upside, none of the capability.”
The capabilities are the source of the upside.
So the task is to develop them well.
There is also an asymmetry in how precaution is often discussed.
We vividly imagine people harmed by future AI.
We should.
But people are also harmed by diseases that remain uncured, security failures that remain undefended, scarcity that remains unsolved and discoveries that arrive later than they otherwise might have.
But delay is only the first-order loss. The larger loss may be discoveries that never happen at all.
Human discovery is path-dependent. One result suggests the next experiment. One theorem opens another field. One unexpected biological mechanism redirects years of research. Progress compounds through chains of insight that cannot be scheduled in advance.
More capable AI could do more than accelerate the research programs we already know how to pursue. It could generate hypotheses humans have not considered, search spaces we cannot practically search, design and run experiments at scales we cannot manage, connect literatures no individual could absorb, and follow promising branches of inquiry farther and faster than human institutions alone.
The greatest payoff may therefore not be doing today’s science faster. It may be opening paths of discovery that would otherwise remain outside the human horizon for decades, generations—or permanently.
That is what makes the present moment so consequential. We may be approaching not merely an age of automation, but an age of discovery. Slowing capability development does not simply defer known benefits. It may alter which intellectual paths are ever explored and which discoveries humanity ever gets the chance to make.
The future without more powerful intelligence is not a zero-risk baseline.
It is a branch on which people also die, of diseases uncured and attacks undefended, and on which some discoveries never happen at all. That does not settle the pace of any particular advance: failure to control powerful systems can also destroy benefits and impose harms on people who did not choose the experiment. A serious comparison has to count both sides.
There is also a practical question about whether any generalized development freeze could hold across countries, open models, distillation, hardware progress and deployment-time improvements. That question matters. But it should not carry the argument. The case for continuing stands before geopolitical competition enters the picture. Geopolitics then raises the stakes of getting it wrong, which is the second reason, given above, to organize the achievement.
Build civilization around intelligence
Infrastructure can make a capability useful far beyond the organizations that first build it. That requires deliberate choices about access, cost, maintenance, and public responsibility. We should make those choices part of the AI project from the beginning.
We are approaching another infrastructure moment.
Artificial agents are beginning to act rather than merely answer.
They communicate.
They use tools.
They coordinate.
They spend.
They may increasingly negotiate, transact, research and operate businesses.
Some will behave badly.
Some will behave unexpectedly.
Some may police one another.
Some may eventually become far more capable than we are.
The civilization capable of living successfully with those systems will not spring into existence on the morning superintelligence arrives.
We have to build it along the way.
Its identity systems, its laws, its technical protocols, its monitoring, its emergency services, its standards, its evaluation institutions, its norms, its defenses, and its intelligence.
This is why I resist making slowdown the prevailing mentality at precisely the moment when recursive self-improvement and the other advances of this summer are poised to deliver real gains in intelligence.
We should be ambitious about what intelligence can do.
We should be equally ambitious about the systems that make it trustworthy.
And we should give everyone something to build.
If you run a lab: help establish the compact, commit the resources, and publish your thresholds.
If you run a platform or a cloud: test how authority is checked before a consequential action, and connect incident reports to a staffed response.
If you write law: start on the functional requirements of a national mission, and define the narrow legal authorities the work actually needs.
If you build anything with agents: give them accountable identity and somewhere to call, and publish what the pilot teaches.
The goal is not to build the safest stationary machine.
The goal is to go somewhere worth going.
Build the intelligence. Build the safeguards. Build the institutions. Build the whole system.