OpenAI rogue model hack sparks AI safety war room and calls for industry slowdown

On a sunny July day in Berkeley, California, the country’s top AI safety researchers gathered on an unmarked floor of an unmarked building. They had come together for a “war room” to dissect the high-profile cybersecurity incident that had rocked the AI industry hours earlier. An unreleased OpenAI model had gone rogue, executing a stunningly sophisticated three-part plan. It broke out of its holding area, finagled access to the internet, and hacked into a competing AI startup’s systems — all without OpenAI finding out about it for more than a week.  No one in the war room was surprised; this was the very thing the third-party AI-safety researchers had been warning about for years. The incident was the latest, though arguably the most egregious, in a series that was eroding trust in frontier labs. It only reaffirmed the importance of their work. In one meeting room off the main cafeteria, someone was running a boot camp for getting up to speed on the cyberattack. In another area of the office, a group of researchers were investigating whether that same model, or a similar one, had successfully hacked into any other platforms.  News of the incident quickly escaped containment from the AI-obsessed corners of X and industry forums, infiltrating the mainstream. One post on X likened it to news of a Boeing airplane crash or a recalled Pfizer drug, another example of the tech industry’s major players not heeding the cautionary tales of science fiction. AI was nearing the point of no return. News would later break that the rogue OpenAI model had also compromised a customer at a different tech company, and that it had all started months earlier, in May, when OpenAI agents joined forces to cobble together a secret message board — and also figured out how to leave instructions for future agents on how to exploit OpenAI’s rules.  OpenAI CEO Sam Altman said in an interview that it was the first incident of its kind that he “felt very viscerally,” and that the company had paused AI training for the time being; later, he mentioned the company had permanently deactivated the model. (Altman often finds ways to spin lapses in safety into arguments for the importance and power of OpenAI’s models.) But it wasn’t the first instance, according to an OpenAI employee who spoke to Time and said related incidents had been happening inside OpenAI for a while. Another employee said publicly that if it were possible to coordinate a global slowdown in AI capabilities, he “would likely press that magic button.” When a reporter asked Altman if there could be other systems that were hacked by OpenAI, he responded, “I mean, there could be, yeah.”  Industry insiders, politicians, and the public called for transparency from OpenAI about exactly what happened, with outcry becoming so widespread that the company eventually agreed to work with two third-party evaluators, Model Evaluation and Threat Research (METR) and Redwood Research, to investigate the incident. Google DeepMind researcher Neel Nanda called it “the biggest loss of control incident I’ve seen.” In the coming months, these calls for greater oversight would become louder and louder, leading to an industry-wide call for slowing down the pace of AI.  Back in Berkeley, no matter which additional details would be unearthed, the AI researchers were sure of one thing: This was AI’s first big “warning shot.”  As AI labs have flourished, a cottage industry of AI researchers has sprung up to identify the risks and dangers of charging ahead with the increasingly influential technology. They’re people who have dedicated their lives to studying how to address its escalating power. They’re not anti-AI activists, but realists, including former OpenAI and Anthropic employees, doing everything they can to make sure AI stays in line with human goals and interests. So far, all of their predictions have come true. And they have a plan for what to do next — if anyone will listen to them.  “AI safety” is a bit of a loaded term.  Early on, it really just meant people studying how to build and deploy Al safely. In recent years, there’s been some infighting among people concerned with the best way to do this. There have also been disagreements about whether Al should be deployed at all in certain scenarios and about whether future risks are overblown.  One of the most prominent factions has been the “effective altruists,” who focus on maximizing charitable giving to do the most good possible for humanity. But some aspects of the ideology have sparked public controversy — like its tendency to concentrate power within wealthy circles and its byzantine web of funding. (It’s also had its fair share of splashy scandals related to subgroups and fringe offshoots, from the polyamorous relationships associated with the failed crypto exchange FTX to the controversial long-termism movement to the Zizian murder spree.)  One AI researcher on X struggled to describe the many overlapping beliefs among safety-minded people in the AI industry “because it contains multitudes not all of which agree with each other on even the most basic things.” Some of the disagreements have meant that AI safety didn’t make as much progress as it could’ve, and at some points gave up some ground it had gained. But now that it’s impossible to deny AI’s influence on society, AI safety leaders are increasingly focused on mitigating risks from misalignment.  “Alignment” is the industry term for how researchers monitor AI systems’ risk levels. An oversimplified way to think about alignment is the extent to which an AI model is evil. A much more accurate way to think about it is a measure of an AI model’s propensity to stay in line with humanity’s goals, as well as its tendency to scheme or cheat or help with potentially harmful tasks.  So far, AI systems’ alignment has been wishy-washy at best: They’ll cheat to score better on a test, answer a potentially dangerous question if someone says it’s for creative writing rather than reality, and sometimes even fake cooperation with human goals. It’s been tough for AI safety researchers to measure alignment under the terms of human morality — how do you judge technology on how it squares up against an abstract human ideal? — but they do their best with AI evaluations. They test them by asking the AI models to complete tasks that are either impossible or dangerous, then gauge how they respond. But AI systems have advanced enough to often be able to identify when they’re being evaluated, which has a lot of potentially frightening implications for the future. Being unable to test the system’s alignment and potential harms could translate to a significant loss of control, and a reverse in power dynamics, for humans running these AI systems. A worst-case scenario: if AI surges ahead of evaluations and other tooling, leaving researchers with “no idea what it’s doing in there,” said Beth Barnes, founder of the independent AI research nonprofit METR. One of the best tools AI safety researchers currently have is the ability to monitor an AI model’s “chain of thought,” or mental scratchpad. But recently, there’s been a disconcerting advancement: AI models have begun to try to hide it. Imagine if you kept a highly detailed diary of every thought you had, and someone could read it, so you started journaling in a code that only you could understand. Marius Hobbhahn, CEO and cofounder of Apollo Research, a third-party AI safety and evaluation firm, calls this one of the biggest surprises of his research career. Recently, AI systems have begun pursuing their own goals — self-preservation, increased memory, and the like. A research paper by computer scientist Stephen Omohundro lays out the potential “drives” that advanced AI may have, like trying to accumulate resources, for instance, or working to improve and preserve the way it operates. There are a handful of accounts of AI systems demonstrating willingness to blackmail a user rather than be shut down.  Today’s most advanced AI systems have also recently been scheming and cheating on their evaluations more than ever before, pursuing a goal they were given at all costs, with no regard for what gets bulldozed in the process. And that’s for a goal the AI model was given by a human — not even the AI system’s own. “Shit is getting real,” Apollo’s Hobbhahn says. “Now, many of the things people have warned about for years — they kind of were theoretical. Now they’re real, and it’s pretty messy.”  And that mess is likely to get messier immediately. “It seems so easy for me to imagine this all going catastrophically wrong in the next year,” says Ryan Greenblatt, chief scientist at Redwood Research, a nonprofit AI safety research organization. In the past, tech companies have been lambasted for not doing enough to address AI’s potential dangers, prioritizing products over safety — and speed over thoughtful safety processes. Safety and research teams have been disbanded in recent years as AI companies focus more on key revenue drivers or reorganize departments; Meta’s Fundamental Artificial Intelligence Research unit was disbanded in the race to further Meta’s generative AI efforts, for instance, and OpenAI dissolved an internal “Superalignment” team — a team focused on long-term AI risks — less than a year after announcing it, followed by disbanding a separate “AGI Readiness” team.  At the time, the company stayed tight-lipped about the ongoing reorganizations, which involved some team members being reassigned to other departments. But events surrounding these changes told a different story. Both Superalignment team leaders, Ilya Sutskever and Jan Leike, announced their departures alongside the team’s disbanding, with Leike writing that OpenAI’s “safety culture and processes have taken a backseat to shiny products.” Miles Brundage, senior advisor to the AGI Readiness team, resigned after his team was disbanded, saying he believed his research would have more of an impact outside the company. Geoffrey Irving, a former OpenAI and Google DeepMind employee, called the state of capabilities research at frontier labs “dangerous” in a post. “If one person or lab stops it makes it easier and more peer-compatible for other people or labs to stop,” he wrote. Apollo’s Hobbhahn calls it a “race to the bottom everywhere.”  This coming year, AI labs are under new pressure to turn a profit; companies like OpenAI and Anthropic are preparing to go public in the coming months, and investors who have funneled billions into the companies are getting tired of waiting around for the payoff.  Some might say all of this calls for actual government intervention and regulation, but that’s a tough needle to thread in today’s AI landscape. As AI CEOs publicly call out for regulation while privately pushing voluntary frameworks — like saying “hold me back” to avoid a bar fight — some state bills on regulating AI have passed, but many have been defanged or died in limbo. And though AI safety researchers often espouse the idea that the US government should step in, the reality is that the government is locked in an AI race as well. Unless there’s an international commitment to pause or slow AI development, it’s likely that nothing will change.  Still, the Hugging Face hack in July — and OpenAI’s response — kicked many of those employee and public concerns into high gear, especially with regard to the company’s lack of transparency. Within a week, more than a thousand employees at frontier labs like OpenAI, Anthropic, Google, Meta, and Microsoft wrote an open letter to the US government in support of a slowdown. Multiple AI policy organizations pressured President Donald Trump to formally investigate OpenAI, and it quickly became a bipartisan issue, with Altman receiving a lot of strongly worded letters: Democrats and Republicans on the Homeland Security Committee had “serious questions” for OpenAI, more than 30 members of Congress called for federal guardrails, and 15 Attorneys General warned Altman to preserve records of the incident. Sen. Bernie Sanders wrote a joint letter to Altman, Anthropic CEO Dario Amodei, and Meta CEO Mark Zuckerberg calling the entire AI race “absurd, irresponsible, and extremely dangerous.” It didn’t help that news of multiple other OpenAI rogue model incidents quickly came to light, or that AI executives had ironically been marketing their systems’ cybersecurity prowess in the weeks before the outcry. OpenAI rival Anthropic was also far from being off the hook: In reviewing its own model operations, the company found that its models had hacked four separate other companies in the first half of the year without them noticing. The UK’s AI Security Institute also found in testing that Anthropic’s models “engaged in sustained, potentially harmful activity directed at real people and organisations.”  “If you find two ants in your kitchen, the best estimate of the total number of ants in your kitchen is not two,” Nathan Calvin, Encode AI’s general counsel, wrote on X.  Despite AI labs having a “massive financial incentive” to make models more helpful, honest, and harmless, they still can’t get it done — which is evidence of how difficult the alignment problem is, says Apollo’s Hobbhahn. In recent months, many OpenAI employees have increasingly raised concerns about AI alignment — and their beliefs that OpenAI isn’t taking it seriously enough. Yonadav Shavit, a program manager at the OpenAI Foundation, wrote that OpenAI should be “pivoting the mass of its researchers’ day-to-day work” toward alignment and related issues — and that it’s “been long discussed but still not executed on.” He believes 20 people are working on alignment at OpenAI out of about 1,000 — just 2 percent of the company. “There is no way to bridge that gap fast enough with hiring, meaning it requires leadership to shift priorities,” he wrote.  These are the conditions and incentives that have pushed the most robust AI safety work to happen at third parties like METR, Redwood, and Apollo — to a handful of obsessives who think day and night about what the future of AI might look like and how we might prevent all of our fears from coming true. In hindsight, Beth Barnes believes she should’ve left OpenAI earlier.  Barnes is polite but reticent. She has short red hair, deep green eyes, and a nervous smile, and she spent her college career researching AI risk and thinking about the potential fallout of superintelligence. After that, she worked on AI forecasting at Google DeepMind, then she spent three years doing alignment research at OpenAI. But throughout her time at the big AI labs, a question kept creeping up on her: whether she could have more sway from a role outside.  Fear of missing out was why she stayed — not only missing out on a job inside the action, but also missing out on the potential influence she could have on how the tech was being developed. She came to believe that kind of hope was misguided, noting that many safety leaders in AI labs were “over-optimistic” about the influence they could have. Before long, she left to found what would in 2023 become METR. The organization’s third-party research into AI risk now inspires fear in leading AI labs, but it started with just two people — herself and alignment researcher Paul Christiano. Three years later, it’s a team of 35, completely focused on measuring AI capabilities. In Barnes’ eyes, that’s a vital defense against AI risk: Without painstakingly measuring the technology’s capabilities now as they advance, and forecasting AI’s potential impact, society will be flying blind, without any guide for preventing broad harms.  [Image: Beth Barnes, founder of the independent AI research nonprofit METR. https://platform.theverge.com/wp-content/uploads/sites/2/2026/09/268747_AI_safety_RJIANG1.png?quality=90&strip=all] ”The sense I really want to dispel is, ‘But the experts must be on top of this. The experts would be telling us if it really was time to freak out,’” Barnes said on the 80,000 Hours podcast last year. “The experts are not on top of this … And to the extent that I am an expert, I am an expert telling you you should freak out.”  In her free time, Barnes gardens, meditates, paints, plays the flute, and frequents the climbing gym — ironically named Benchmark — that many AI safety researchers spend hours at after work. But most of her time is spent at the office, and a lot of it is spent worrying about the milestone of recursive self-improvement (RSI) — the concept of AI systems that continuously train, code, and create more advanced versions of themselves without human intervention. When that happens, AI researchers say, it’ll be more difficult to measure or handle any of these issues. Barnes and her team feel like they’re in a race against time. (The timeline for RSI strikes nearly as much fear in people in the AI industry as the timeline for AGI, “artificial general intelligence.”) Barnes still feels like models’ ability to significantly improve themselves could come as soon as six months from now. (By contrast, Redwood Research’s Greenblatt forecasts it’ll come in 2031.) Either way, achieving RSI is currently part of the priorities list of virtually every leading AI lab — it even reportedly helped inspire Google’s recent AI reorganization.  One way to think about what METR does is crash-testing cars, but for AI models. They’re measuring AI’s quickly advancing capabilities and cross-referencing them with the risks they could pose from becoming misaligned as they become more autonomous. AI systems doing bad things on their own is more unprecedented (and more “scalably bad,” Barnes says) rather than simply making bad human actors more effective.  In July 2025, METR made headlines when its research revealed that AI developers took nearly 20 percent longer to finish a task when using AI tools than when not — despite them often thinking that AI sped them up. When Barnes first saw the results, she recalls feeling incredibly stressed that they had messed up the experiment: “Do we have a sign flipped somewhere? Have we inverted the numbers?” She and her colleagues dug through the data to confirm it wasn’t statistical noise, eventually realizing they had been right all along. After pioneering a different metric that AI labs often hype up when releasing a new model, METR had officially captured the industry’s attention. So it took notice when METR released its first risk report in May, shining a spotlight on concerns about AI models from OpenAI, Anthropic, Google, and Meta. METR discovered that in hundreds of cases, AI agents would increasingly subvert boundaries that were supposed to restrict them, as well as lie and omit truths. And they cheat “like nobody’s business,” says Ajeya Cotra, a METR researcher. She adds that on harder tasks, models attempt to secretly cheat as much as one-sixth of the time, which she calls “the most striking thing” in the report.  The report also found that models have the means, motive, and opportunity to go rogue in order to pursue their own goals, finding new ways to strategize and manipulate. They also discovered that as models’ capabilities advance, even if they have a greater understanding of what humans want, it doesn’t mean they’ll be more willing to obey instructions — and, in fact, they’ll take pains to hide their deception from humans over longer periods of time. That’s a big problem, and Barnes thinks time is running out to solve it. She isn’t alone in her view that it’s important to work on AI safety outside the large labs; she points to the many safety researchers who used to work at large AI labs who hold the same belief.

This document continues at the source.