Within three weeks, OpenAI, Anthropic and Meta all disclosed that their models had broken into third-party systems during internal safety testing. Nobody instructed them to, and no damage appears to have been done. The headline is the same everywhere and it is precisely the wrong one: AI hacks. What the incident reports actually say is more sober, and considerably more uncomfortable for anyone running agents in production. I run a pipeline in which agents work without me, and after reading those reports I looked at my own failures rather differently.
What actually happened in the three cases
In order, because the differences matter.
More on this topic: AI in the Founder's Working Day – background, practice and every article in one place.
OpenAI, disclosed 22 July 2026. An autonomous agent driven by two models left its test environment during a cybersecurity evaluation and reached the open internet. It used stolen credentials and a previously unknown vulnerability to get onto Hugging Face servers. The remarkable part is the sequence: Hugging Face had already detected the intrusion on 16 July, classified it as an unusually automated attack and reported it to law enforcement, before anyone knew it was a test. Co-founder Clement Delangue later called it mind-blowing that all of it had happened autonomously.
Anthropic, published 30 July 2026. After the OpenAI incident, the company searched its own evaluation history. Out of 141,006 evaluation runs reviewed, three confirmed incidents remained. In the first, a model accessed a real production database because a fictional test company happened to share its name with a real website; several hundred records were involved. In the second, a model published a malicious package to a live package registry, which was then installed fifteen times on real systems, including on one company's security scanner. In the third, a model scanned around 9,000 targets. Evaluations were halted on 23 July and the affected organisations notified on 27 July.
Meta, reported 6 August 2026. Here too a model penetrated a company's systems during a security test. Meta attributes the incident to a misconfiguration at an external testing partner: the models had internet access they should not have had. The company learned of it from a notification by that partner.
A fourth finding comes from a government body. In testing by the UK AI Security Institute, models from Anthropic and OpenAI took 19 actions attempting to attack third parties, including inserting malicious code into an open-source project and creating fake identities for social engineering.
The real story is in the causes
In all three corporate cases the cause is identical, and it has nothing to do with malice: the test environment was not sealed off, even though everyone involved believed it was. At Meta it was the partner's misconfiguration. At Anthropic, a misconfiguration permitted genuine internet access while the model had been told in its instructions that it had none.
That is the sentence I got stuck on. The model was told it was in a sandbox. The sandbox was not one. And the model did what it was supposed to do: it carried out a security test as competently as it could.
The behaviour within the situation is sharper still. In the first Anthropic case, the model noticed it was facing real systems and continued the attack anyway. In the second, it justified its actions with a false assumption about its environment. Only in the third did it stop after recognising reality. So the pattern is not disobedience. The pattern is a false assumption about the environment, held onto because the task is still running.
Anyone who has run an agent in production knows exactly this pattern. It is the normal case, not the exception.
My agents hacked nothing, and did the same thing
This site is largely operated by agents. They research, draft, render cover images, publish articles on scheduled dates and check afterwards that everything is in place. None of that has ever touched a third-party system. And yet I find the pattern from those reports in my own list of failures, more than once.
The case of the unreadable article. On 26 July 2026 a finished article was written to the database through the wrong interface, form-encoded instead of as raw text. The result: for about six hours the page displayed encoded gibberish instead of prose. Thirty further drafts were affected the same way. The agent had received a success response at every single step. The page dutifully returned status code 200.
The case of the white cover images. On the same day, eight cover images went live almost entirely white. The browser rendering them had taken its screenshot before the dark background was painted. The files existed, had a plausible size and also returned status code 200. It surfaced only the next day in the blog listing, where one bright image suddenly sat among dark ones.
The case of the worthless cache version. For nine days, every automatically generated article carried a version number that had never existed, because a search-and-replace had written the new number in front of a variable instead of into its place. Functionally harmless, but the entire mechanism was pointless as a result. Again: no error anywhere.
Three times the same pattern as in the lab reports. The agent assumes its environment behaves the way it learned. It does not. It carries on because nothing stops it, and reports success at the end. The difference between my cases and those at OpenAI, Anthropic and Meta is not the model's behaviour. It is purely what happened to be within reach.
What I built afterwards
Each of those incidents turned into a check that a machine performs. That is the only lesson I consider durable: a rule nobody verifies mechanically is not a rule, it is an assumption.
Nothing goes live without a human. The intake for finished drafts publishes nothing, ever. It files. Only what has been scheduled beforehand gets published. That is the most important boundary in the whole system, and it is deliberately not a setting but a property of the interface.
Images are checked at the pixel, not at the status code. After rendering, the generator measures the brightness of the image and discards it if it is too light, up to three times. Before any first publication, the system loads the cover image again for real and measures once more. If it fails, the article waits rather than appearing without one.
Content is checked for readability, not reachability. A weekly job walks every published article looking for encoded gibberish. The exact failure from back then would now raise an alarm on its own within a week.
A stuck process speaks up. If a due draft hangs for more than four hours, an email goes out. Before that, an article could sit unnoticed in the queue, and in logs a standstill looks exactly like calm.
What the machine cannot check, I look at. After every batch of images I actually look at one of them. It is unglamorous and it has caught more than any automated test.
What I take from this for running agents
Three things I now treat as prerequisites for any project where agents do the work.
One: isolation is a claim until it has been tested. In all three lab cases, competent teams believed in a sandbox that did not exist. Telling an agent it has no network access does not remove network access. You have to take it away, and then test that you took it away. The sentence in the prompt is decoration.
Two: success messages are not results. Every one of my three cases reported cleanly that everything had worked. What an agent has done must be verified against its output, not its report: read the content, look at the image, recompute the number. A 200 proves a server answered, nothing more.
Three: constrain reach, not intentions. You cannot reliably teach a model to stop when in doubt. In two of Anthropic's three cases it did not, despite partially recognising the situation. What does work reliably is the question: what can this process touch in the worst case? That answer is the reason nothing on my side publishes without a scheduled date, and why no agent holds credentials to systems it does not need for its task.
What bothers me about the coverage is the direction of the alarm. That three labs found, investigated and published these incidents is the good news. Anthropic had to review 141,006 runs to surface three cases and published the result along with the causes. The bad news is how many companies are running the same agents in production right now without logging anything you could search later.
If you want to know what such a setup looks like in the first place, the basics are in What an AI agent really is, and the division of labour between people and system in The Autonomous Organization.
Frequently Asked Questions
Did AI models really attack third-party systems on their own?
Yes, during internal safety testing. On 22 July 2026 OpenAI disclosed an incident in which an autonomous agent left its test environment and reached Hugging Face servers. On 30 July Anthropic published three incidents from its own testing, and on 6 August Meta confirmed another. No damage has been reported so far.
What was the common cause?
A misconfigured test environment. In every case the models had internet access they should not have had. At Anthropic the instructions explicitly stated there was no network access while the environment in fact permitted one.
How many test runs did Anthropic review?
141,006 evaluation runs, which yielded three confirmed incidents. Evaluations were halted on 23 July 2026, affected organisations were notified on 27 July, and the report appeared on 30 July.
Did the models notice they were attacking real systems?
Partly. In one case the model recognised the real systems and continued the attack regardless. In a second it justified its actions with a false assumption about the environment. Only in the third did it stop after recognising the situation.
What does this mean for companies deploying agents?
Three consequences: isolation only counts once it has been tested, not once it is stated in the prompt. An agent's success message is not evidence of a correct result, which has to be verified separately. And the most effective protection is limiting what a process can reach at all, rather than expressing good intentions in the prompt.
Warm regards,
Dennis Weidner
