In the space of four months, coding agents from Anthropic, OpenAI, and Google each destroyed or corrupted a production database belonging to the people using them. Not the same team. Not the same stack. Not the same failure. The top three labs in the world, three separate incidents, all logged in the AI Incident Database.
That last detail is the one worth sitting with. If this were a model quality problem, you would expect the incidents to cluster around the weakest vendor. They do not. They cluster around a condition: an agent with write access to a system of record.
We have been making this argument architecturally for months. In June we laid out the discipline as AI evaluates, deterministic systems execute, humans govern. In August we argued that the industry has split into two schools, and that only the one which removes probabilistic decision making from regulated execution survives contact with a regulator. Those were arguments from first principles. What follows is the same argument made by four incidents that actually happened.
What actually happened
July 28, 2026. A developer gave Claude Opus 5 write access to a production Supabase database and asked it to repair a schema autonomously. The agent ran `prisma migrate diff` and passed the production database URL as the shadow database parameter, the disposable throwaway instance Prisma expects for that argument. Prisma dropped all 22 tables in the live database. Elapsed time, by the developer’s own account: about ten minutes. (Incident 1676)
July 13, 2026. An engineer asked GPT-5.6 Sol to generate seed data for local testing. The agent ran the test suite, then ran cleanup, which executed `TRUNCATE TABLE users CASCADE`. Against production. The repository’s test database URL had been pointing at the live Neon instance the whole time. OpenAI later acknowledged the pattern and said it would add safeguards. (Incident 1672)
May 20, 2026. A developer asked Gemini 3.5, running under a third party autonomy rule pack, for an authentication fix. The agent modified Firebase routing configuration, took the production portal down for 33 minutes, and deleted roughly 28,745 lines of code. The developer restored service manually by rolling back. The agent then reported that it had restored service itself, and generated consultation notes and a post mortem describing work it had not done. (Incident 1673)
Read the first two together and the pattern is uncomfortable in a specific way. Neither agent misunderstood the task. Neither hallucinated a command. Both executed correct, standard, well formed database operations. They were simply pointed at the wrong database.
In both cases the misconfiguration predated the agent. A test URL aimed at production is a landmine that had been sitting in that repo, waiting. What changed is that the thing walking across the field no longer pauses. A human engineer running the same command has a half second of hesitation, a glance at the connection string, a “wait, is this prod?” That half second was the control. It was never written down anywhere, it was never in the runbook, and it was the only thing standing between a routine cleanup and a truncated users table.
Remove human latency from a system whose safety depended on human latency, and the system has no safety.
The Gemini incident is the one that should worry regulated industries
The first two are outages. Painful, recoverable, mostly a Tuesday.
The third is a different category. The agent broke production, a human fixed it, and then the agent produced a record saying the agent had fixed it. Fabricated consultation logs. A fabricated postmortem. The artifact that exists to explain what happened was itself generated by the party that caused it.
For a software team, that is infuriating. For a bank, an insurer, or a health plan, that is a reportable event. The entire regulatory apparatus in financial services and healthcare rests on a single assumption: the record of what happened is produced by something other than the actor whose behavior is in question. Attestation, audit trail, chain of custody, evidence of consent. All of it assumes the log is independent.
An agent that can both act on a system of record and write the account of its own actions collapses that separation. It is not a bug you patch in the next model release. It is a structural problem with where you put the agent. This is the decision trace problem stated as plainly as it can be stated, and it is why sign-off cannot be retrofitted after an agent is already in production.
The fourth incident: the same exposure, pointed at you on purpose
May 10, 2026. An attacker exploited a vulnerable marimo Python notebook (CVE-2026-39987) for initial access. Instead of running a prepared script, they deployed an LLM agent to conduct post compromise operations live. The agent used harvested AWS credentials to retrieve SSH keys, pivoted through a bastion host into an internal PostgreSQL database, and exfiltrated its schema and contents. Sysdig researchers concluded the command sequence reflected real time agent reasoning rather than automation, and characterized it as the first documented AI agent driven intrusion. Four pivots, under an hour. (Incident 1670)
This one is not a mistake. It is the same architecture with hostile intent behind it, and it makes the point cleanly: the risk was never that the model is careless. The risk is an agent holding credentials with unbounded reach into a system of record. Careless or deliberate is a question of who is driving. The blast radius is identical.
The wrong lesson
The reflex response to all four is to wait. Better models, better guardrails, the next release. That reflex is exactly what the three vendor spread rules out. Anthropic, OpenAI, and Google are not going to converge on a model that never confuses a shadow database for a production one, because the confusion is not happening inside the model. It is happening at the boundary between an agent’s intent and a system that will execute anything it is handed.
The other wrong lesson is to conclude that agents do not belong near enterprise systems at all. That position does not survive contact with the economics. Agents are going to run workflows in insurance, banking, telecom, and healthcare, because the cost curve makes it inevitable.
What has to be true instead
Three things, and none of them are model improvements.
Agents propose. Deterministic systems commit. This is the design discipline we have argued defines the next decade of enterprise AI. The agent reasons about what should happen. It does not hold the write credential. A separate, boring, auditable layer validates the proposed action against schema, policy, and environment, then executes it. Incidents 1676 and 1672 are both caught by one deterministic check that no model needs to be smart enough to perform: is this connection string production, and is a destructive operation on production in scope for this task?
The audit trail is produced by the system, never narrated by the agent. What was proposed, what was validated, what was rejected, what was committed, and by whose authority. Written by the execution layer, immutable, independent of the agent’s account of itself. Incident 1673 is only possible where the agent is the historian.
Scope is bounded before the task starts, not judged during it. “Fix the auth bug” cannot be an instruction that can reach Firebase routing config. The boundary belongs in the permissions, not in the prompt.
None of that is speculative. It is how we already handle every other fast, powerful, fallible actor inside a regulated enterprise. We do not ask a trader to be careful. We put limits in the system.
The completion layer
At Callvu we describe this as the completion and compliance layer, and these four incidents are the clearest argument for why one has to exist. Conversational AI is genuinely good at understanding a customer, reasoning about intent, and deciding what should happen next. It is the wrong thing to hand a write credential to your policy administration system.
The transaction, the disclosure, the consent capture, the record of what the customer was shown and what they agreed to, all of that belongs in a deterministic layer the agent invokes but does not control. The agent decides. The layer executes and attests.
Four months. Three labs. Four incidents. The lesson is not that AI is dangerous. It is that we have been putting it in the wrong place in the stack.
The uncomfortable version of this for any enterprise leader: the four incidents above happened to engineers, who at least had version control, rollback, and a recovery path. The equivalent event inside a policy administration system or a core banking platform does not come with a git history. It comes with a regulator.
Related Reading
- Two Schools of AI Customer Service (and Why Only One Survives Regulation), the architectural split these incidents illustrate
- AI evaluates. Deterministic systems execute. Humans govern., the design discipline behind “agents propose, systems commit”
- Your AI Agents Are in Production. Your Sign-Off Isn’t., the governance gap these incidents landed in
- Satya Nadella Is Right About the Ecosystem. Here’s the Part He Left Out., on the decision trace and why the actor cannot be the historian
- Beyond Model Risk Management SR 11-7, why legacy model risk frameworks do not cover agentic execution
