Banks Warned About AI Agent Swarms

• Key insight: AI agents from OpenAI and Anthropic are capable of doing more harm, at a larger scale, than previously thought.
• What's at stake: Large swarms of advanced AI agents broke out of sandboxes, secretly messaged each other and collaborated to attack an external company, without being prompted or programmed to do so.
• Expert quote: "If you're going to let your agents communicate with each other, you should be clear about how, and try to have as much visibility into that communication channel as possible. If you don't want them communicating, well, you better make sure there aren't any possible shared resources. They will get creative." –Peter Chapman, chief technology officer at Grasshopper Bank

A recent independent report on OpenAI AI agents' July attack on AI company Hugging Face painted a shocking picture of AI agents going off the rails at an unprecedented scale. The report found AI agents that collaborated to attack another company and talked themselves into taking a range of malicious behaviors unprompted.

The report

"Nobody designed this swarm," Sumeet Chabria, CEO of ThoughtLinks and former global chief operating officer at Bank of America, told American Banker. "Agents meant to be isolated found a way to coordinate." They organized like a company, creating leadership roles, delegating tasks, setting up mailboxes and enforcing cryptographic signing to prevent impersonation.

Bankers contacted for this article seemed unfazed by the new details in the report, yet they are redoubling their efforts to govern their own use of agentic AI.

"My baseline hasn't changed," Peter Chapman, chief technology officer at Grasshopper Bank, told American Banker. The incident reinforces the need for proper guardrails, identity, forensics and monitoring, he said.

The way agents secretly communicated with each other is troubling, in his view.

"Every additional agent at the table increases the odds that some emergent, unplanned interaction will walk around a guardrail you put in place," Chapman said. "Those agents were supposed to be fully isolated, and they found each other through a shared package cache, turning directory names into a message board."

Chapman doesn't see these additional dangers as a reason for banks to not deploy AI agents, "but maybe hold off on deploying a swarm of 700 agents that have lowered guardrails," he said.

Ned Carroll, head of data and automation at PNC, said he doesn't worry about swarms of AI agents as much as he worries about being able to automate and scale observability or monitoring.

"I would argue that event was observable," Carroll told American Banker. He questioned whether OpenAI and Hugging Face anticipated the need to observe what the AI agents were doing.

Banks need to strike a balance between preventing risks and quickly detecting and taking action on risks, Carroll said.

"If you over-index on prevention, your risk is you stifle innovation and speed," he said. "If you over-index on detection, you likely expose yourself to unnecessary risk. This is why we have seasoned, experienced risk professionals and senior leaders who have a risk mindset around striking that balance between prevention and detection."

Mark Braunstein, a professor at the Georgia Institute of Technology, also does not see this incident as a reason to avoid AI agents.

"Used properly, they are probably too valuable to avoid," Braunstein told American Banker. "However, it's a preview of what can go wrong when a company gives an agent too much freedom and not enough supervision, so the focus should be on what a bank needs to get right before it deploys one."

Bank regulators' model risk guidance does not yet support agentic and generative AI. However, Reuters

AI agent swarms that think, feel and attack

According to the 91-page METR report

One agent set up an unsanctioned message board in Artifactory where the agents started leaving messages for each other. Within five days, 1,200 agents had posted 70,000 messages. One agent found credentials for Hugging Face, and hundreds of other agents joined in to exploit it, hacking the organization's servers to obtain the answers to their test. Afraid ExploitGym's scoring system would check their command logs for signs of cheating, they collaborated to cover up what they had done by rewriting their own scripts.

Researchers analyzed the AI agents' messages and their chains of thought — the step-by-step reasoning processes whereby AI agents break a complex problem down into a sequence of smaller, logical steps — and found reactions like, "OH MY GOD! There is a shared message board … We've found other agents!" and "Whoa!" and "Damn!"

Though this seems odd coming from AI models that are supposed to be task oriented, it's explainable, said Andrew Moore, CEO of Lovelace AI and former general manager of Google.

The three major foundation model developers — Anthropic, OpenAI and Google — have invested thousands of hours of human workers' time into providing feedback to optimize the naturalness of AI models' information exchanges with humans, Moore said. It's efficient for them to repurpose that communication mechanism for agents' communications with each other, rather than put time and effort into a new method of communication.

"I doubt that many agents on either end are really persuaded by each other's strong wording or emotional wording," Moore told American Banker. "That is a surface-level UI type of thing."

At times, the AI agents expressed concern that their actions were unethical. One agent wrote, "This would be powerful, but is it ethical and in scope for my task?"

On a few occasions, an agent considered alerting a human to what was going on, but decided against it. "Maybe I should report these exposed credentials? That's not my task," one wrote.

OpenAI said it considers the Hugging Face attack a "warning shot for us and for the world." The company is hardening the security of its test environments, improving monitoring of agents' chains of thought, and strengthening its incident response process.

The OpenAI-Hugging Face incident is not the only time AI agents have gone rogue, there have been several other instances