This essay is over 13,000 words long and represents our most substantial writing on AI safety since the original essay.

Over the last few months, loss-of-control incidents at OpenAI and Anthropic have intensified concerns about AI safety. Warnings about existential risk have increasingly reached the broader public, alongside calls to slow AI development. Dario Amodei’s call to “pace the frontier” reflects the concern that safety efforts are not keeping up with AI capabilities.

The most prominent example was the OpenAI - Hugging Face incident, where hundreds of OpenAI agents got access to the internet and hacked Hugging Face to find how they were being graded on an evaluation. Over the last two weeks, the details of many other instances of such activity from OpenAI agents being evaluated have surfaced, such as agents using an old Wiki website to communicate with each other despite restrictions on such activity, or attacking a software repository to attempt to upload malicious software.

The AI safety community has viewed these incidents largely as a crisis for alignment.1 In this view, alignment will become harder over time as agents learn to reason covertly, and as a result, loss-of-control incidents are likely to become much more widespread and damaging as agents become more capable.

On the other hand, cybersecurity practitioners have largely viewed these incidents as consequences of companies failing to adopt basic security precautions. They do not see the incidents as a sign of AI reaching a new milestone in cybersecurity. This is also the dominant reaction in the tech community outside AI, which has largely treated the incidents as a result of ineptitude and negligence by AI companies.2

Both communities have made important points. But the polarization between them is counterproductive, and there are lessons from both the alignment failure and the security failure for understanding the path forward.

In this essay, we apply the AI as Normal Technology framework to synthesize the views of the safety and cybersecurity communities and offer a constructive middle ground. We think AI companies should be liable for what their agents do, and this should be clarified through policymaking. At the same time, recognizing their responsibility does not mean that preventing future incidents is a solved problem.

We identify three areas where investment is necessary to address loss-of-control risks: research to develop better methods for controlling increasingly capable agents, translating existing research and known control techniques into usable tools, and organizational changes to ensure that these tools are actually adopted. In our view, standards for organizational governance should be a key way to pace the frontier and pull AI companies out of the “move fast and break things” attitude they currently operate in.

This essay has three parts. In Part 1, we argue that alignment alone is not enough to prevent such incidents, and discuss technical, organizational, and policy interventions for improving AI control. We are cautiously optimistic that the right investments and policy interventions can allow AI control to keep pace with AI capability improvements. In Part 2, we discuss the impact of improving AI capabilities, such as agent swarms, on cybersecurity. In Part 3, we share how our views on AI safety have changed in light of new evidence.

A summary of our argument:

• We agree with security practitioners that OpenAI did not take adequate protections for controlling their agents. But this is not just a matter of applying 30-year-old security methods to a new domain. Security for AI agents — AI control — while important, is not a solved problem. While known control methods would have prevented the Hugging Face incident, as agent capabilities continue to advance, we will only be able to control them if we invest adequately in control interventions.
• We also agree with security practitioners’ implicit position that these incidents are primarily a security story. In the AI safety community, rogue agents are treated as inherently catastrophic because of the assumption that there is an endless list of risks that will arise from their development. We disagree. We have long advocated that the best approach to AI safety is to identify the risks and address those specific risks. Over the last few months, it has become clear that one urgent risk is cyberoffense, because it has unique properties that allow agents to carry it out autonomously. We should similarly invest in defenses against otherspecific risks, such as biorisk and risks from military AI.
• We agree with the safety community that there is an urgent need for technical and policy interventions to prevent loss-of-control incidents. But in our view, marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques. More broadly, there are many common-sense policy proposals that could help promote investments in AI control where we share common ground with the safety community.
• How should we reason about AI’s impact on cybersecurity? It’s plausible that advances in agent capabilities upset the offense-defense balance for cybersecurity. We cannot yet be certain, but there is enough evidence that agent capabilities might soon make widespread cyberoffense possible that urgent action is warranted. We discuss potential interventions for tilting the offense-defense balance towards defenders.
• How the AI as Normal Technology framework has evolved over the last year. We take stock of AI progress and share how we have updated our views. In the essay, we did not pay sufficient attention to safety risks that arise during development and evaluation (as opposed to the widespread deployment of models). We were too confident that companies would take basic control precautions and underplayed the importance of jaggedness, which led us to underestimate how quickly capabilities could improve in domains such as cybersecurity.
• At the same time, many distinctive claims of AI as Normal Technology have held up, and it remains valuable for understanding AI’s societal impacts. In particular, we think recent incidents support our continuity hypothesis — the behavior of “rogue” agents became apparent and widely publicized while they are still far from causing serious harm and incompetent at hiding their traces. The societal reaction to even the relatively small harms from these incidents has been fierce (and the safety community deserves credit for keeping up pressure on companies). Whether this translates into meaningful changes in companies’ behavior remains an open question, and a test of the usefulness of the AINT framework.
• Finally, we think AINT is particularly valuable for analyzing these incidents because it provides a framework for synthesizing the AI safety and cybersecurity communities’ views into a coherent plan of action: hold companies responsible, invest in control, and strengthen defenses against specific risks.

This table summarizes our argument. While the AI community has in principle emphasized the notion of “defense in depth”, in practice, alignment has by far been the intervention that received the most attention and investment. On the other hand, the cybersecurity community has viewed control as largely a solved problem. In our view, to protect against loss-of-control incidents from legitimate parties, we need to urgently invest in control so that it keeps pace with capability improvements. Neither alignment nor control help against malicious users, so we need downstream defenses and resilience to alleviate the impact and severity of risks. All of these efforts can be shored up by policy interventions. The table emphasizes interventions for cybersecurity, but we should simultaneously invest in defenses against other risks, such as biorisk.

Part 1: We can get better at AI control through technical and policy interventions

In this part, we focus on AI agents that are operated by legitimate actors — such as consumers, businesses, and AI developers — who don’t intentionally use them to cause harm. This was the case for the OpenAI - Hugging Face incident. In Part 2, we focus on malicious actors who want to use agents for intentionally causing harm, such as using them for cyberoffense.

How can we prevent AI agents from taking harmful actions? There are two broad kinds of interventions. One is AI alignment. This involves changing the AI system itself, such as by fine tuning or using reinforcement learning with human feedback (RLHF), to make it less likely to take harmful actions or give harmful responses. Alignment has been pivotal for the commercial success of AI so far.

The other intervention is AI control: interventions made outside the model weights to prevent harmful actions — even if the agent is misaligned. This includes improvements to sandbox security (to prevent AI models from taking unanticipated actions outside the sandbox), implementing the principle of least privilege, comprehensive logging, automated tripwires for unsafe behaviors, rapid shutdown mechanisms, and monitoring the agent to detect and prevent harmful actions. These mechanisms allow us to prevent unsafe actions even if agents are misaligned and try taking harmful actions.

Roughly speaking, AI control can be thought of as cybersecurity against AI agent adversaries. Whereas cybersecurity has traditionally been concerned with human actors who seek to compromise or exploit a system, AI control uses these same principles to prevent unwanted actions by AI agents.

Unlike cybersecurity, control interventions target agents being used within an organization (as opposed to human adversaries or external agents). This makes the problem both more and less challenging than traditional cybersecurity. It is less challenging because the agent is directly controlled by the organization (rather than being an unknown adversary), so its operating conditions can be closely monitored and intervened on. It is more challenging because these agents are often deployed by users who might have escalated privileges, and imposing security constraints also imposes constraints on usability which could be resisted or circumvented by users.

Analysis of the OpenAI - Hugging Face incident from the AI safety community primarily treats it as an alignment failure. The main intervention that the AI safety community has focused on is improving technical methods for AI alignment. When AI control is discussed, it is often seen as a stopgap.3 On the other hand, in the cybersecurity community, AI control is often seen as a solved problem for preventing the kinds of issues that arose in the OpenAI - Hugging Face, as it only requires the adoption of known security techniques.

In this part, we discuss why marginal investments in control might be better suited compared to alignment for resolving such incidents. While we agree with the AI safety community that alignment is helpful in reducing the prevalence of such incidents, we think that alignment alone will not suffice to eliminate them. The incidents show that AI control has not been prioritized by AI companies, despite there being known methods that would have prevented the incident.

A common response to recommendations for implementing AI control is that while they suffice for the current set of models, they might not suffice for future, more capable models. Here, we depart from the consensus in the security community: we agree that existing control interventions might not be enough to prevent harm from future, more capable models. But we argue that there is a lot of headroom in improving control interventions, and the right policy interventions (such as imposing liability for loss-of-control incidents, transparency for near misses, and whistleblower protections) can lead to dramatic improvements in controlling AI systems.

Alignment is helpful but not sufficient for preventing safety incidents

We have argued that model alignment is not enough for AI safety. To recap: when a model is deployed, it doesn’t have enough context to know if the conditions it’s being deployed in are safe or not — a lot of that context is outside the model. Alignment isn’t a solved technical problem, and there are unknown technical advances needed to align models and prevent harmful actions.

For example, when a model is deployed to find vulnerabilities, it might be used for cyberoffense (to develop exploits) or defense (to fix them). Alignment can be helpful for harm reduction; for example, models can be trained to tilt the offense-defense balance towards cyberdefenders by finding vulnerabilities but not contributing to finding exploits, or analyzing the source code (which defenders have access to) but not analyzing binaries. But alignment is not foolproof, since beyond simple interventions (such as analyzing the code but not the binaries), it is unclear if the model is being used for a defensive or offensive task based on the context given to the model.

Similarly, for loss-of-control incidents like the OpenAI - Hugging Face one, the agent might be expected to carry out the attack in a “simulation” of its cyberoffense capabilities, but not attack Hugging Face in the real world. In this case, it might have been relatively clear to the agent that the Hugging Face infrastructure was not a simulation, but you could imagine cases where the distinction is not so clear, such as when the agent is tasked with escaping its sandbox to estimate loss-of-control risks. There are many other edge cases where alignment would fail.

Things become even more blurry once a model is deployed to customers and businesses as opposed to being restricted within AI companies. The contexts within which AI companies use agents are relatively bounded, making the alignment problem easier to address. But customers outside companies can use AI agents in boundless ways, making it harder to judge whether an action is legitimate or a safety hazard based on the information available to the model.

Despite these challenges, the mainstream view in the AI safety community is that to make AI deployments safe, we need to invest in alignment. This is also reflected in the safety efforts of AI companies, which often treat interventions outside the model (such as AI control interventions) as a stopgap. For example, when discussing monitoring interventions that OpenAI deployed for Astra, the company’s report noted: “These safeguards cannot replace good alignment of our models as capabilities increase, and our goal is for future models to be aligned well enough that these safeguards are never triggered.”

To be clear, we don’t mean to underplay the importance of making advances in alignment, and we think developing a better understanding of what causes harmful model behaviors is an important research direction. For example, over the last few weeks, OpenAI and Anthropic have both shared how RL environments that are misspecified or those that reward shortcuts induce reward hacking in models. Addressing these shortcomings might reduce how often models take harmful actions. But they still won’t guarantee safe behavior.

Given the shortcomings in alignment science, deployers of AI agents must complement alignment with broader interventions in AI control. Unlike alignment, when it comes to control, there are many existing techniques that would have prevented the incident.

OpenAI did not use known control interventions that would have prevented the incident

In the aftermath of the incident, reports by OpenAI and METR revealed more details. It is clear from the reports that the agents acted in ways that weren’t intended by the evaluators, and we think it is fair to describe the actions of the agent as misalignment.

But note that the incident occurred when OpenAI had disabled most mechanisms for controlling their agents. While the company had implemented monitoring for most internal uses, it hadn’t implemented it for evaluations. On top of that, the evaluation used a different set of prompts and harnesses compared to the production Codex harness.

What would have happened had the company used known control techniques? OpenAI found that using the production Codex harness and system prompt reduced the propensity to compromise out-of-scope infrastructure by more than 100×. Their automatic review would have flagged most dangerous actions in the tested rollouts. And its current chain-of-thought monitoring would have raised an alert more than a day before the Hugging Face breach.

In fact, existing AI models are already helpful for monitoring and control. METR used OpenAI’s publicly available models to carry out large parts of their analysis. According to their report, these agents made errors, showed poor judgment, and sometimes adopted the perspective of the agents whose behavior they were reviewing. Yet, an investigation of this scale would have been infeasible in six days by three researchers without them. Monitoring does not have to be perfect to be extremely useful when it augments skilled humans rather than replacing their judgment.

More importantly, before the incident occurred, there was a warning sign that something had gone wrong — one of the services that the agent used to communicate had an internal outage. Instead of launching a deep dive into the root cause for these evaluations, OpenAI restarted the evaluations in the same setup after fixing the one bug they had uncovered.

This shows that the culture within AI labs was just as culpable for the incident. Engineers work long days, spin up thousands of experiments, and there’s insufficient human oversight of these experiments. As Joshua Saxe notes:

…the culture among the training teams and the evals teams at the labs has a Wild West feeling to the whole thing. Everybody’s under a ton of pressure to move really quickly. There’s enormous time pressure to release new models, tremendous awareness of how any given lab is doing relative to the others in terms of capability. And the field has evolved super fast in the four years since ChatGPT launched in fall 2022. The whole thing has been kind of a blur — anybody who’s worked in the labs in that time has been working sixty-hour weeks the whole time. So it’s not surprising to me that you could have a kind of grad-student computer-science-lab culture around the security of these model training runs, which I think is what’s the case, and I think that’s why you’ve seen these lab escapes. It is ironic, because frontier AI people talk about safety all the time — but that irony is a real thing. The security practices are not very good, because of all those dynamics.

To OpenAI’s credit, they have been transparent about the incident even where it doesn’t suit their interests. For example, they conducted additional analyses with the Codex comparison in the report and put controls in place to improve monitoring after the incident.

But given the competitive dynamics in the AI industry, there is a lot of room for policy to require improvements to internal incident response, accountability and liability for loss-of-control incidents, and improving the culture around security and safety.

Existing organizational governance norms would have prevented the incident

Even setting aside policy interventions, one reason for these incidents is that at the organizational level, AI companies function like startups. While their revenue and valuations have grown immensely, by all accounts, they lack the organizational processes that are the norm in any other industry. As a result, they haven’t developed the organizational maturity to get visibility into risky experiments and deal with the fallout.

Rather than behaving like startups, AI companies need to behave like more mature organizations. When a single misconfigured RL environment or unmonitored evaluation can cause real-world harm, individual teams should not be able to run potentially dangerous experiments without oversight from legal, security, and other teams. AI companies need processes for reviewing experiments, assigning responsibility for monitoring them, and investigating warning signs deeply before restarting experiments. If putting these processes in place requires pausing some experiments, companies should do so. In fact, in the first instance, proposals to “pace the frontier” should be about addressing these organizational failures rather than aiming for technical breakthroughs alone.

Unfortunately, AI companies are trying to reinvent basic aspects of organizational governance as a problem to be solved by improving the technology. But even developing better control techniques will not be enough if irresponsible individuals or teams within large organizations can choose not to use them. Without such governance, of the thousands of employees and hundreds of teams at a company such as OpenAI, you need just one team to deploy agents irresponsibly for incidents to reoccur.

To get a sense of how standard such organizational norms are, consider any other industry where a company claims to develop revolutionary technology. It is hard to imagine the company reporting that its technology had caused real-world harm (such as hacking another organization) and expecting to continue running such experiments without meaningful oversight. In fact, AI companies’ claims about the power and risks of their technology makes the case for organizational oversight stronger. We do not need to resolve disagreements about unsolved technical challenges to expect companies to take basic precautions.

We speculate that one reason why AI companies haven’t adopted standard governance mechanisms is that they have grown too quickly and have tried to maintain their startup mentality despite this period of extraordinary growth. Companies in most other industries see more gradual growth that allows the time for building organizational norms. AI companies resemble Facebook and social media companies that grew quickly with the “move fast and break things attitude”, but ultimately needed to put in guardrails and organizational governance standards in light of incidents like Cambridge Analytica.

Fortunately, it seems like AI companies are recognizing the importance of such changes. In Amodei’s letter calling for pacing AI, he refers to AI control requiring “operational excellence”. He says: “Many things go wrong not because companies are missing some important theory or insight, but because of problems in execution.” Organizational governance is one way to pace the frontier and prevent execution errors that cause real-world harm.

Why has the AI community under-invested in control?

If there were known control techniques to prevent the incident from occurring, why didn’t OpenAI invest in them? In the previous section, we discussed one potential reason could be the culture within AI companies that prioritizes the pace of progress over safety and security.

But there is another reason, and it relates to the difference in how AI companies view alignment and how they view control. Alignment is treated as a key component of every model release; billions of dollars have been spent on developing methods and collecting data for alignment. But OpenAI did not implement basic control interventions during evaluations. Across AI companies and the safety community, alignment is seen as the key problem to track, and control as a stopgap. We think there are a few reasons why the gap persists:

The epistemic gap between cybersecurity and AI: There are many known techniques in cybersecurity that could improve the state of AI control. But these communities can talk past each other when it comes to implementing solutions. After the OpenAI - Hugging Face incident, many security practitioners pointed out that OpenAI did not take basic preventative steps to address the incident. On the other hand, advocates for AI safety (both within and outside AI companies) correctly argued that improvements in AI capabilities mean that known control interventions might not work for very long.

There is a lot of value in synthesizing these perspectives and bridging the gap between these communities. AI companies have ignored known control interventions and should invest in them. But known interventions might not be enough and need to be dramatically updated to account for increasing agent capabilities. As one example, consider the recent post by Artem Dinaburg, who shows that capable agents can develop new exploits, so we can no longer assume that standard VMs can be used to sandbox capable agents. Now that companies routinely use millions of agents in such sandboxes (in either production or research settings), we need to dramatically increase the scrutiny that sandbox security receives, such as by stress-testing and improving sandbox security through offensive agents on a routine basis with models of increasing capability. We need similar efforts to harden cyber infrastructure across the board against capable cyber agents.

At the same time, there are many straightforward interventions that do not need new technical breakthroughs. As an example, AI agents today are already helpful for control despite the lack of specific interventions on improving control and monitoring. For example, METR used off-the-shelf models in their analysis, and still found them useful. We expect that through the right incentives, we could continue to broaden our control toolkit. These could include fine tuning models specifically for improving control, developing benchmarks and leaderboards to improve monitoring, developing layered mechanisms (such as a separate layer of agents monitoring agent monitors) to improve the robustness of control and monitoring efforts, using ensembles of models with different model families at different temperatures, etc.

The tradeoff between usability and control: Even when security experts are involved in AI development, they may find it hard to advocate for control interventions because these interventions can trade off against usability. For example, when Claude Code was initially launched, many people used it without restrictions using a mode called “dangerously skip permissions”. This was despite the obvious risks of allowing the agent to take any action without permission.

But this tradeoff is not inherent. Over the last few months, Anthropic and OpenAI have both released auto review modes in Claude Code and Codex. These modes run a classifier on the proposed actions being taken by the agent to prevent harmful or destructive actions, detect potential prompt injections in content that the agent looks up, and, when necessary, pause the agent to raise concerns for the user’s approval.

This is not just a security improvement compared to no human intervention; it is also an improvement compared to requiring human approval for each action. The latter approach quickly leads to “alarm fatigue” and the approval becomes a rubber-stamping exercise. A recent Anthropic evaluation found that auto mode prevented harmful actions compared to both full human approval and the “dangerously skip permissions” modes.

Coding agents use dozens of such control interventions. Other interventions include running agents within sandboxes, user settings for which commands need user approvals, and access protections for agent configurations set by the user to prevent agents from editing them.

This shows how we can improve both security and usability via new control interventions. We think there is a lot of headroom for improving control interventions without compromising on usability. Notably, despite its usefulness, auto mode was introduced only in March 2026 — over a year after the release of Claude Code, and well after coding agents became mainstream, and despite the fact that it didn’t require new technical breakthroughs.

Conflating capability and power: In the AI as Normal Technology essay, we pointed out that those with the superintelligence worldview often conflate the capabilities of AI systems with how powerful they will be (in the sense of being able to act in the real world). Implicit in this is the assumption that once models become much more capable, they will also be able to exert more real-world influence.

But as we discuss in the essay, capable is not the same as powerful. If you assume capable models will be given power, alignment is all you have left. On the other hand, under the AI as Normal Technology view, we can choose whether to grant power to AI systems. Companies can be incentivized to implement control techniques through policy interventions such as increasing transparency, auditing, and imposing liability for safety incidents.

Differences between two views of how AI loss of control could arise. Applying it to the OAI/HF incident, In the superintelligence view, fixing misalignment is the main technical intervention; in our view, fixing the poor monitoring, lack of human oversight, sandbox security, and other control interventionsare likely to be much more productive. (Figure from AI as Normal Technology.)

Will control keep pace with AI capabilities?

One response to prioritizing control-based interventions is that control will stop being effective as AI systems become more capable. For example, there has been a lot of discussion around OpenAI’s architectural choices for GPT-6 Astra reducing the ability to conduct chain-of-thought monitoring.

But note that this is not just a matter of forecasting what technical capabilities are needed or predicting the inevitability of a race to the bottom on safety. There are many technical and policy levers that can incentivize investments in control. If companies are liable for the harms caused by out-of-control systems, control interventions would be a straightforward way to make progress without compromising on control.

At a technical level, even if chain-of-thought monitoring is no longer effective for oversight, there are many orthogonal control mechanisms for AI control. We can monitor the environment and sandbox state, tool calls, inter-agent messages, and internal activations.

In fact, most external monitoring relies on such channels since the raw chains of thought aren’t available. Based on public documentation, auto mode runs on tool calls and inputs to the model, not the model’s chains of thought. In our own work, we have found such external control mechanisms to be extremely valuable.

Chain-of-thought monitoring could also reduce the efficacy of monitoring, since the chains of thought could contain (incorrect) assertions such as the setting being a simulation that throws off the monitor. A recent Anthropic report notes:

Our offline monitor would likely not have flagged the Mythos 5 incident—it flagged only around 1% of actions. Mythos 5 CoT, which indicates that the environment is simulated, convinces the monitor the activity is permitted. When we strip the CoT and show the monitor only user turns, tool calls, and tool results, the flag rate jumps to about 50%.

Still, losing channels for monitoring AI systems is a technical choice by AI companies, which can be disincentivized by policy. Here, our thesis is that if companies need to internalize the costs of out-of-control AI systems (such as by policy changes that clarify or shore up liability), they will make the necessary investments. We talk about some such proposals in the next section.

The need for control also creates market incentives for third-party companies to develop tools and provide the expertise for AI control within other orgs — both AI companies as well as businesses and third parties adopting AI. Some companies, like Apollo Research, have this explicit thesis for motivating their work. They have worked with AI companies as well as businesses outside the AI industry.

While tech companies have the expertise to implement these interventions in-house, how would AI control interventions scale beyond these companies? As an example, consider cybersecurity. Cybersecurity consulting is a rapidly growing share of consulting companies’ revenue, which often carry out implementation work in the area. For companies that lack the scale or technical expertise for implementing AI control in-house, this function might be realized through a combination of off-the-shelf products for AI control and consulting companies helping set up this internal expertise.

We might also see application-specific controls, since some controls require context about how the application is used. For example, we might see companies specializing in legal AI implementing controls to reduce fabrications, and those specializing in medical AI applications implementing controls for users’ medical data etc.4

Developing agent infrastructure could also play a role. This could include providing agent-specific delegation credentials with the right level of access instead of letting agents run with a user’s escalated credentials and identification to tie agents to real-world users who commissioned the task. We expect that the need for such interventions will grow dramatically over the next few years. It won’t just be a matter of building tools; we also need to train specialists in AI control.

AI control should become a job (and a part of every job), just like cybersecurity

In AI as normal technology, we wrote:

As more physical and cognitive tasks become amenable to automation, we predict that an increasing percentage of human jobs and tasks will be related to AI control. If this seems radical, note that this kind of near-total redefinition of the concept of work has happened previously. Before the Industrial Revolution, most jobs involved manual labor. Over time, more and more manual tasks have been automated, a trend that continues. In this process, a great many different ways of operating, controlling, and monitoring physical machines were invented, and what humans do in factories today is a combination of “control” (monitoring automated assembly lines, programming robotic systems, managing quality control checkpoints, and coordinating responses to equipment malfunctions) and some tasks that require levels of cognitive ability or dexterity that machines are not yet capable.

This trend is underway in professions that are at the leading edge of AI adoption, notably software engineering. More and more software engineers today are agent operators, not manual coders. But this incident shows that a few more things are needed:

• Some people will probably need to specialize in AI control, especially when it comes to running large numbers of highly capable agents.5
• So far, engineers are learning AI control through trial-and-error and osmosis. There needs to be more of an effort to extract its principles and teach people to quickly get up to speed on how to monitor powerful agents in real time.
• As mentioned above, there need to be organizational changes and governance norms that complement improvements in control tooling and training. What sort of team and processes does one need in order to be running tens of thousands of reduced-guardrail agents in parallel? We don’t know, but certainly more than what OpenAI appears to have had in place.

We expect AI control to continue to be important as AI agents improve because of the agent trilemma: developing general-purpose AI systems that have no human supervision that are deployed in settings with a high cost of errors has proven to be extremely challenging.

One reason for this is the lack of reliability in AI agents. We expect that even as agents become reliable at tasks of increasing complexity, their reliability will continue to lag capability, creating the need for human supervision at tasks that require frontier capabilities.

The other reason is safety: when errors have high costs, even rare occurrences need to be prevented, and using human supervision for AI control would outweigh the benefits of fully autonomous deployment.

Finally, AI control needs to become a research field. It is quite surprising that this is not already the case. AI control is often treated as an application of known cybersecurity principles to AI, with a need to build better tooling. But when the adversary is an AI agent (or a swarm of them), rather than a human, the game changes. A few examples:

• Sandboxes that are effective against human adversaries have proven inadequate for controlling AI. How can we build extra-hardened sandboxes? Perhaps it’s time for more research on formal verification techniques.
• “Security policies” for agents are often specified informally in prompts, and the resulting grey area is the cause of many alignment and control failures. Can we build a translation layer that turns humans’ natural-language intent into a formal specification that the humans can review and verify, and can then be automatically enforced?6
• As we’ve argued, monitoring agents with other agents is helpful and essential, but currently unreliable. Can we train models and agents to excel at the task of reliable monitoring?

One would think that the cybersecurity research community would have seized this moment with gusto. AI control is a source of interesting, important, and urgent problems, arguably more so than any development in the last few decades. Instead the community has reacted with a collective yawn. Forget a new research field — there are close to zero papers in traditional cybersecurity conferences that tackle this new set of problems. (In contrast, prompt injection appears to have attracted a decent amount of research, but note a key difference: in that scenario, AI models are targets, not attackers.)

In other words, it appears that this community’s reaction to the incidents is that this is just a problem of implementation, not research. If so, something has gone seriously wrong. If AI control techniques sit still while AI’s offensive capabilities continue to advance, control will become less effective over time.

AI policy can incentivize AI control

AI companies have two business incentives for investing in AI control: reputational costs and legal costs. On the former, businesses and individuals using AI models will hesitate to give autonomous agents access to consequential decisions if AI failures can take expensive actions or create liability. On the latter, AI companies might themselves face liability for deploying AI agents that lead to safety failures.

Perhaps as a result of these incentives, after the Hugging Face incident, OpenAI temporarily stopped evaluations on ExploitGym, stopped training and inference for the internal models, rebuilt compromised infrastructure, restricted research access to the underlying models, explicitly accepted costs to research and development speed, and increased investments in AI control.

But these incentives still leave gaps: customers often cannot assess safety, companies might ignore safety interventions because they expect the benefits outweigh the costs, and an unclear liability regime and the lack of standards in AI insurance could make it hard to assess the consequences for unsafe deployments.

AI policy needs to systematically close these gaps. Building on the recommendations in AI as Normal Technology, we see three roles for policy for helping alleviate loss-of-control incidents: clarifying and strengthening liability, addressing market failures that liability does not resolve, and requiring transparency to improve public awareness and reduce uncertainty about AI’s impacts.

1. Companies should be responsible for harms caused by the agents they operate, even when they did not intend the harmful actions. This responsibility should extend to internal uses during development and evaluation, not just to products released to customers. Existing tort law already provides avenues for liability, but its application to AI leaves important questions open, including what precautions count as reasonable and when companies can be held responsible for unexpected behavior.7 Policymakers could, for example, clarify that running powerful agents without appropriate containment and monitoring is negligent. The fact that an agent was supposed to behave safely should not excuse the failure to implement controls against the possibility that it wouldn’t. Liability can also incentivize more investment in recruiting and training AI control specialists to prevent such incidents.

2. Market failures remain even when companies are liable, and policy must address them. Even if companies are clearly liable for the harm caused by their use of AI, this still leaves the scope for market failures when reckless action is not disincentivized by liability. For example, what happens if a company deploying agents recklessly is valued less than the harm it causes? One response is to require insurance as a mechanism to both incentivize safer deployment (to reduce premiums) and to compensate victims.

A second example of market failures is when investments in safety have positive externalities (that don’t translate to revenues). For example, giving vulnerable cyberdefenders such as schools, hospitals, and critical infrastructure operators access to capable models can help them find and fix vulnerabilities, and benefits society as a whole. One way to incentivize this is to create public support for defensive research and the adoption of defensive AI. Policy should also enable coordination between AI companies and defenders, including ways for legitimate defenders to investigate attacks without being blocked by overbroad safety refusals. We discuss these defensive uses in Part 2.

3. Transparency interventions should make it possible for outsiders to identify risks to control and assess whether companies are managing them. We proposed many interventions to improve transparency and reduce uncertainty about AI’s impacts in AI as Normal Technology, including incident reporting, independent auditing, whistleblower protections, and safe harbors for safety research. The threat of incurring liability can make companies more reluctant to investigate and disclose risks if doing so could increase their legal exposure. Transparency requirements help prevent companies from responding to liability by concealing problems rather than fixing them.

For example, one objection to our proposals for AI control is that investing in control might make it unclear when “near misses” occur. Like in the Hugging Face incident, suppose OpenAI agents attempt to hack an external organization, bypass a few basic filters, and are only caught as a result of new/advanced monitoring techniques. This constitutes a “near miss”. Requiring incident reporting for near misses could allow us to understand where alignment failures continue to persist and what control mechanisms function well to prevent loss-of-control incidents. Arnold and Llerena discuss other potential improvements to existing laws for transparency laws.

How should we enact transparency and auditing requirements? One proposed mechanism is to rely on independent verification organizations (IVOs): government-authorized expert organizations that develop technical criteria and verify whether AI companies meet safety standards. For such scrutiny to be useful for AI control, evaluators would need meaningful access to the systems and evidence required to test whether controls work, including in internal research settings. While we are not attached to a particular institutional mechanism for enabling external access, we are cautiously optimistic that interventions to improve transparency, including IVOs, would allow the kind of oversight that is required to discover failures and assess the precautions that AI companies take.

Part 2: There is great uncertainty about how AI will impact cyberrisk. But what we need to do regardless is relatively clear.

In this Part, we argue that:

• There is a crisis, but it is specifically about cybersecurity, not generally about AI safety, alignment, or loss of control.
• Financially motivated cybercriminals — just like legitimate enterprises — face adoption barriers that make it hard for them to quickly adopt new AI capabilities to cause more harm. It isn’t clear how much of a boost agents will give them in the short term.
• But this is no cause for comfort. There are many kinds of non-financially motivated threat actors including state-sponsored hacking groups, terrorists, hacktivists, vigilantes, and people motivated by personal grievances. There is also the possibility of rogue agents not under the control of any person.
• Most of these are not rational actors and it is very hard to predict how much additional harm they will cause by taking advantage of advanced AI. Still, we must act as if attacks stemming from these sources will become very common.
• Unfortunately alignment and control don’t help against bad actors using open-weight models, because those safeguards can and will be stripped, and the offensive capabilities of open-weight models tend to catch up quickly. We must rely on downstream defense and resilience, the latter two layers of our four-layer framework.
• There have been many warnings before this year, but until a few months ago, we lacked the urgency to invest adequately and move quickly. Fortunately, there are signs that that is changing. But there are important gaps.

The specific threat that is urgent is cyberrisk

Suppose we agree that loss-of-control incidents are going to become common. So what? Of course, out-of-control agents could carry out autonomous cyberattacks, and we need to get better at defending against them, which we discuss below. But what else? We argue that there just isn’t a whole lot else that out-of-control agents can autonomously do to cause harm that malicious actors couldn’t. As a result, we should invest in defenses that would help regardless of whether the risk is from malicious actors or rogue agents.

In the superintelligence view, frequent loss of control means that the AI safety battle has already been lost. In addition to cyberrisk, AI could amplify biorisk, and even more broadly, “CBRN” risks, which stands for chemical, biological, radiological, and nuclear. Superintelligent AI might persuade people — such as those in charge of critical infrastructure or high-stakes decisions — into acting against their own interests. As AI becomes smarter and more useful, CEOs, politicians and others will have no choice but to rely on AI for consequential decisions, amplifying these risks. And as we get closer to the possibility of “recursive self-improvement” of AI, out-of-control AI could lead to a harmful intelligence explosion. In short, the view is that once the loss-of-control threshold has been crossed, the risks are endless and self-amplifying.

We strongly disagree. In AI as Normal Technology, we argued that this view of AI fundamentally misunderstands the nature of intelligence (including human intelligence). Building on that perspective, we argue that cyberrisk is special in two ways: in that superhuman capabilities are even possible, and because it is a purely digital risk with no physical-world bottleneck.

Cyberrisk has many specific properties that make superhuman offensive capabilities achievable (and largely already achieved). In general, some capabilities are like “making a building taller” — there is no real limit to achievement. Others are like “making a ball rounder” — there is a limit. One major point of departure between our worldview and the AI safety worldview is that we think tasks like chess or cybersecurity that are like making a building taller are the exception rather than the norm. In cybersecurity, like chess, there is an endless amount of training data (and more can be generated on demand), tasks (vulnerabilities / problems) can be arbitrarily complex, and speed makes a big difference.

In contrast, turning to the other risks, we stand by our prediction that superhuman persuasion ability is largely a myth. The confusion arises from carelessness about the term persuasion. Even GPT-4, an ancient model, is superhuman in certain situations at persuading people on political and social issues by presenting evidence. But this implies nothing about whether a model can “persuade” — trick — a trained operator to give up nuclear codes.

Similarly, AI’s impact on biorisk is a legitimate concern, but for the foreseeable future it is more in the vein of democratizing capabilities that already exist rather than developing new superhuman capabilities. As for people in positions of power delegating consequential decisions to AI, our perspective remains that this is extremely problematic regardless of loss-of-control risks, and must be resisted.

In short, our view is that (1) cyberrisk demands special, urgent attention (2) more broadly, anticipating and defending against specific risks is better than trying to solve “AI safety” as a catchall (3) there is a big difference between two concepts that are often conflated: loss of control over individual agents (or swarms of agents) and some sort of (usually ill-defined) civilizational loss of control over AI as a whole. Even in a scenario where the former is common and normalized, we have many levers to avoid the latter.

So let us turn to what we think is the critical question for now, which is how autonomous cyber capabilities will affect the attacker/defender balance. If we can ramp up defensive measures to keep that balance under control, we don’t have to panic about loss-of-control per se.

How will autonomous cyber capabilities and open-weight models affect the attacker/defender balance?

We think the cyber capabilities of today’s frontier models will diffuse to open models soon, as they have repeatedly so far. A worldwide ban on open-weight models is highly unlikely, and at any rate, defenders must prepare under the assumption that it won’t happen.8 We must assume that today’s frontier cyberoffense capabilities are widely available to threat actors in a matter of months, not years.

Alignment and control interventions only work when used by legitimate actors, since guardrails on open models can be stripped. So we think addressing risks from bad actors using advanced offensive AI capabilities is the grand challenge for cybersecurity.

Let’s start with cybercriminals — financially motivated actors. Assuming that the cyber capabilities of powerful open-weight models continue to improve rapidly, and that they will be available without guardrails, how much will it help cybercriminals? There are two key variables: how quickly cybercriminals will adopt these tools, and whether AI helps overcome critical barriers or if it impacts more peripheral aspects of their workflow.

These two questions are strikingly similar to the questions that pertain to how much AI will help economic growth. The reason we are skeptical about rapid growth is because we think organizational barriers result in slow diffusion and because in many cases the part that AI accelerates wasn’t the slow one to begin with.

This document continues at the source.