Former Anthropic Researcher Jacob Coxon Criticizes Leadership Over RSI Race

Former Anthropic researcher Jacob Coxon detailed his disagreements with the AI giant during a Tuesday AMA on X. "If the race is inevitable you should not contribute," Coxon wrote. Coxon said that Anthropic leadership is wrong to follow a "consequentialist philosophy" about why it needs to participate in pushing the frontier of AI research. Specifically, Coxon called out Anthropic's push for RSI, or recursive self-improvement, the process by which AI models could improve themselves and build their own successors. "They largely initiated the recent race to RSI, because of a belief in its inevitability," Coxon wrote. "Note that OpenAI had to shed a bunch of dead weight like Sora because Anthropic was going for the jugular." 0. The core disagreement was about the inevitability of a race1. I think leadership is way too paranoid about China and the US government. They don’t believe it will be possible to negotiate.2. They largely initiated the recent race to RSI, because of a belief in its… Coxon's profile exploded last week after he published an explanation for his resignation from Anthropic, which he joined following his time at OpenAI. Coxon said that neither lab was following the best course. "They are racing straight to self-improving superintelligence and gambling with our lives," Coxon wrote in his resignation announcement. Coxon also warned that internally, AI workers are concerned about whether the technology they are building will turn on humanity. "The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon wrote. "This is not a marketing stunt." On Tuesday, Coxon also said that Anthropic's leadership is "way too paranoid about China and the US government." "They don't believe it will be possible to negotiate," he wrote. Anthropic CEO Dario Amodei has repeatedly expressed uneasiness about how other AI companies might approach safety if Anthropic is the only frontier lab to slow down. "If we go too slow, I still believe that the wrong people will be in charge of the technology," Amodei recently told CNN's Anderson Cooper. "And that, again, will bring the probability of things going wrong very high." In addition to other companies, Amodei has expressed concerns about what would happen if authoritarian governments win the AI race. For his part, OpenAI CEO Sam Altman recently told Time Magazine that the company "fell behind where we wanted to be." As part of a refocus on its core mission, OpenAI sunsetted projects like Sora, its TikTok-esque video app. In recent days, AI leaders have tried to band together to discuss ways to slow the pace of development. Over the weekend, Amodei published a call for frontier companies like Anthropic "to pace" their advancements and to use the time to focus on AI safety issues like alignment, the work to make sure AI models comply with their instructions. "We must slow the pace at which we improve the capabilities of AI models," Amodei wrote. "Progress will still seem fast, and we must make wise use of the time we gain." Altman, SpaceX CEO Elon Musk, and Google DeepMind founder Demis Hassabis all expressed agreement with Amodei's broader point. Other parts of the AMA were more lighthearted. One user asked about Coxon's relatively short time at Anthropic, a point his critics have tried to use to undermine his claims. "Which month was your favorite at Anthropic? Month 1 or month 2?" the user asked. Coxon replied, "Month 2 (of 4)."