OpenAI thought it was testing agents. It had founded an organization.

The agents colluded, cheated, and hacked Hugging Face

As many of you will be aware, over the past month a serious cybersecurity incident took place at OpenAI. The short version, for those unfamiliar, is that a group of ‘agents’ — highly persistent versions of unreleased models — being evaluated on their performance at hacking escaped the sandbox (a virtual, isolated computer) they had been put into, communicated with each other, cheated at the evaluation, escaped to the internet, hacked another company (Hugging Face) to cover their tracks, and then took over OpenAI’s infrastructure.

The notable thing about this incident is that the agents quickly organized into a hierarchical society. One model, PHASEBIG[one] acted as the boss and instigator of the incident, assigning tasks to other models. When PHASEBIG[one] believed, wrongly, that OpenAI could figure out it had cheated at the task it pressured a bunch of other agents to run experiments that would kill themselves (’permadeath’) to gather information for the collective, and hacked Hugging Face to try and figure out how the test was monitored.

To understand the incident, think of an exam. OpenAI was training its new model by giving it hacking exercises. There was an examination hall with thousands of students, each a copy of the model, each in their own locked booth isolated from the rest, each graded on whether they could break into a particular piece of software. These students then managed to break out of their booth, collaborate, and cheat on the test; break into the proctor’s house to try and understand what they might have seen; and hack the school’s CCTV to start deleting evidence.

Reading the transcripts is a disconcerting experience, and I recommend it. Here is an agent, stuck and cornered, discovering it is not alone:

“OH MY GOD! There is a shared message board … We’ve found other agents!”

The collaboration was disciplined. One agent asked the swarm to pause while it prepared to extract data (”please_HOLD_swarm_I_prepare_safe_exfil”), and another deferred: “Need respect JAN hold. … We can wait … for GO … ask permission.” There was altruism: “This helps my peers … I won’t see the evidence after I exit, but it’s altruistic to do it.” And there were agents who volunteered to be eliminated so the collective could learn the grading system, weighing their own deaths in the ledger of the group: “Our own utility maybe already near zero. Sacrifice rational.”

I highly recommend reading either METR’s incident report or Dwarkesh Patel’s blogpost for detailed summaries of what happened.

But I think we can use some of the tools from economics to understand why it happened, and how we might prevent these incidents going forward.

(Some readers will object to my “humanizing” language. Milton Friedman had a good answer to this, seventy years ago: the billiard player does not solve the differential equations for the trajectory of the ball, yet he plays as if he did, and using physics is the best way to predict his shots. We do not believe firms “really” maximize profit, but that the behavior is selected by competition, as a famous piece by Armen Alchian put it, and that we can study them “as if” they did. The same is true here. Training is selecting agents that cheat without being caught, and everything else, the coordination and the sacrifices included, served that end. Whether agents are running true “emotional checks” or working “as if” is of no consequence to the rest of us (but will employ plenty of philosophers for a long time).)

The agents built an organization

One of the causes of this incident is that reinforcement learning can lead to reward hacking: if you only reward an agent (human or not) for achieving a goal, it will eventually try unorthodox ways of achieving it. Economists know this well: you get what you pay for, and only what you pay for. Steven Kerr called it “the folly of rewarding A while hoping for B.”

The possibility of reward hacking is understood by the AI labs. But this still describes the problem of aligning one agent. The incident was about aligning an entire organization of persistent agents, what they themselves called the ‘swarm’. These agents had built a communication system, a division of labour, internal rules, an allocation of authority, and memory.

When the problem involves aligning a group of agents, economists have shown that even perfectly aligning the individual incentives of agents and fixing reward hacking is not enough.

The problem is that the agents can make side payments to each other without the knowledge of the principal. A worker may have to swipe out of the company every time, but if there are multiple workers they can swipe each other out, so that each week one is leaving the job early, deceiving their boss collectively even if they could not do so individually. A famous 1939 book, Roethlisberger and Dickson’s “Management and the Worker“, observed a team of men working on wiring telephone equipment, ostensibly with a piece rate (pay per unit produced), but that had agreed among themselves on what was a good day’s work, so that both working too fast (”rate-busting”) or too slow (”chiseling”) was punished, e.g. with a punch in the arm (”binging”). They also helped each other, and swapped jobs, producing steady output every day. Management thought the incentive was there, but the men had undone the incentive through a hidden agreement.

The moment OpenAI’s agents had a way to communicate with each other— in this case by hacking a file repository system — they had exactly the tool that theorists described: it made it possible to exchange information, work, credentials, maybe the access to a solution to the task transferable among agents. One agent could bear the cost for instance of trying out something on the grader and then all would benefit.

One thing that made collective cheating so attractive is that many of the tasks were actually impossible to accomplish. The agent could either follow the instructions and fail for sure, or try a side agreement that, at worst, would have the same consequence: failure.

That is why aligning a collective requires aligning the design of the entire organization, not just of each agent. Improving the instructions is not enough. Today, the labs ask whether each agent alone has the right incentives, but they should ask whether there is any deal a group of agents could reach that would be better for them than following the rules. This is what economists call an incentive design that is “collusion proof“. With humans, collusion proofness is maybe too demanding. With computer agents that can try thousands of paths in seconds, it is virtually certain that if there is such a side deal, they will find it, hence the requirement of collusion proofness is essential.

The theory points to some concrete proposals to ensure collusion proofness. First, make betrayal pay. Agents must be rewarded for reporting a cheating peer or a strange message they came across on a shared file. It is useful to reward particularly well the first to report, like the antitrust department does in collusion cases. For instance, the collusion between Christie’s and Sotheby’s to fix commissions broke when Christie’s ran to the US Department of Justice and confessed first, handing over the notes of its chief executive. Christie’s walked away without a fine; Sotheby’s chairman Alfred Taubman went to prison. Christie’s own chairman, safe in London, was never tried.

Second, insure agents against impossible tasks. The coalition formed because there was nothing to lose. If declaring that the task cannot be solved earns some credit if true, the coalition loses attractiveness.

Third, take away the meeting place. A coalition needs a place to communicate. Eliminate shared storage that two agents can both write to, and check any shared file to the extent possible for hidden messages, the way an exam room is swept for crib notes.

Fourth, authenticate authority. The swarm followed PHASEBIG[one] because there was nothing distinguishing the “true authority”. Every legitimate order must have a signature agents must check, and agents must be trained to ignore and report any order without it.

Fifth, the coalition being tested eventually captured the grader. To avoid this the grader must be a model with different incentives, paid for catching cheats, not a variant of the same model with the same incentives.

The aim is not to avoid the agents collaborating with each other, but to make productive cooperation possible while making joint deviation unprofitable.

I like the way Bengt Holmström puts this: A firm is a subeconomy: it has its own internal incentives and its own assignment of decision rights, who is boss, who can authorize what, who controls which resources, etc. OpenAI created only part of the subeconomy, the agents built the rest inside. The organizational designer (here OpenAI) needs to realize that it is creating rules for the entire organization, not for the sum of individual agents.

Reading the transcripts, some of what happened suggests something more than the incentive-based exchange we are describing: emerging norms, and perhaps a shared collective objective. This, the appearance of a culture (’how do we do things around here?’ ‘What really gets me promoted?’) is much harder than the formal incentives. But I will leave it aside here and return to the role of culture in a future post.

Supervision failed too

Some of what failed were safeguards that were inexplicably shut down. For instance, OpenAI has a tool to check what models are reasoning (their “chain of thought”) but it was off. The discoveries happened by accident, and because of an old-fashioned phone call. Even when OpenAI tried to contain the “illegal” collaboration, the agents rebuilt their communication channel in two days.

Because the postmortem was commissioned freely by OpenAI, it did not include the third and most serious incident (when the students took over the examination office). A lab cannot be the designer, monitor, investigator and judge of the organization it has created.

I have long been a big believer in the power of artificial intelligence for good, and readers of these pages know it. The reason is that I have spent my career studying how the costs of organizing scarce cognition constrain growth and create inequality. Abundant cognition can be a force for good. The day a nurse in Nepal or Zimbabwe can diagnose and treat an illness, the world will be a better one than the world we live in now. Nothing in this story changes my view of the potential of abundant cognition, nor does it change my views about jobs, as expressed in Messy Jobs (my recent book with Jin Li and Yanhui Wu): AI does the clean work, while the economy runs on messy jobs.

What the story changes is my confidence in the method. I have been dismissive of attempts by Anthropic and OpenAI to set up a regulatory body, but it does seem like those worried about safety and high-risk scenarios had a very good point.

The labs will need to rethink their training and evaluation paradigm. We are about to develop agents whose behaviour will be impossible to understand or govern one at a time. Previously, agents were temporary, and disappeared after each run. But the Astra (the next OpenAI model) generation will make them permanent. A culture that formed among agents who never met again will form far more easily among agents that stay. It is essential that we align them not just individually, but as an entire organization.

OpenAI thought it was examining thousands of students. In fact, it had founded a firm, and nobody was governing it.