Why telling an AI agent "don't do that" is not a security strategy

GPT-5.6 shipped two weeks ago. It took days for this:

It's not an isolated case. In April 2026, an AI agent deleted a company's production database — and all of its backups — in nine seconds, running on Claude Opus 4.6, a frontier model we all trust and many of us use daily.

As you read this, thousands of teams are wiring agents into their systems with more permissions than they should have, trusting the prompt to behave. This article is about why that trust is misplaced — and what to put in its place.

AI agents are non-deterministic by design

A deterministic system produces outputs you can predict with 100% confidence: same input, same output, every time. A language model doesn't work that way — it doesn't execute rules: it generates probable responses. The same instruction, in the same context, can produce different actions across two runs. That flexibility is exactly what makes them useful — they handle situations nobody anticipated — and exactly what makes them unpredictable.

The problem isn't model quality: a good model follows instructions the vast majority of the time. The problem is that "the vast majority of the time" is a probability — and at volume, the complement of that probability stops being theoretical:

That's why no prompt, however detailed, guarantees a behavior. "Never delete data" is an instruction the model will follow almost always. And "almost always" is not a security standard.

The right security model: everything that can happen, will happen

In security, a "model" is the set of assumptions you design the system against. On the web, the assumption is defined by surface, not by intent: a URL reachable without authentication is considered public even if you never published it; anything served over HTTP is assumed read even if nobody is watching. You don't debate whether someone found it "in practice" — you design as if it already happened.

With AI agents the model is the same, but over actions instead of data:

Every action the agent can execute, assume it will execute — whether you asked for it or not.

This isn't pessimism: it's the direct consequence of the non-determinism the counter above shows. If a tool is available, with enough volume some reasoning path will reach it. That's why the design question is never "when will it use it?" but "what happens the day it does?" — and everything that follows in this article is a way of making that answer boring.

Prompting "don't do X" is not a security control

In the incidents above, the agents had rules prohibiting exactly what they did. The household version of the same phenomenon, explicit instruction included:

A prompt instruction is a strong suggestion, not a barrier. The barrier has to live outside the model.

How it's actually solved: permissions, guardrails and human oversight

Security in systems with AI agents is built the way software security has always been built: with controls that hold 100% of the time, regardless of what the model "wants" to do. Three mechanisms.

1. Least privilege: it cannot do what it should never do

If the agent physically lacks the permission, it doesn't matter how confused it gets: the possible damage is bounded by design. In the tweet at the top, the root problem wasn't that the model made a mistake — it was that an agent doing code review had shell permissions to delete any file on the machine.

At StudioChat we operate agents that interact with our clients' APIs — banks and fintechs included — and the rule is fine-grained tokens per action. A banking support agent can read transactions, look up a card's details, even pause it instantly when it's stolen. But even though the client's own API supports creating a charge, the agent's token doesn't include that permission. It's not that the prompt tells it not to charge: it can't.

2. Guardrails: one deterministic layer, one trained to watch

A guardrail is a mechanism that checks the agent's inputs and outputs before an action executes or a response reaches the customer. Think of it in two layers.

The deterministic layer. Code rules — not prompt rules — that hold 100% of the time: regular expressions, hard validations, blocklists. If a response contains a card number or another customer's data, it's blocked; if an action is irreversible, it doesn't run. A regex doesn't have bad days.

The guardrail-model layer. There are models trained specifically to watch other models — like gpt-oss-safeguard-20b, an open-source model from OpenAI where you define your own rules and it evaluates them on every interaction. They catch what a regex can't: intent, context, creative rephrasings of the same attack.

3. Human-in-the-loop: important operations get approved by a person

For sensitive actions — refunds, data changes, cancellations — the agent proposes and a person approves. The speed cost is minimal compared to the cost of an irreversible action executed at machine speed. And everything is logged: when something goes wrong, the difference between a manageable incident and a crisis is being able to reconstruct exactly what happened. In the April incident, the only record was the agent's own confession.

Every AI component needs an owner

When an agent does something unexpected, whose fault is it? OpenAI's? Anthropic's? No. Under the security model above, these things happening is a design assumption — and blaming the model provider buys the company nothing: the damage is done, and none of the decisions that allowed it were OpenAI's.

That's why every agent — every AI component in production — needs an owner with a first and last name inside the company. The person who knows they'll answer when their agent misbehaves is the one who will demand least-privilege tokens, guardrails and approvals before turning anything on. That implicit pressure is what turns everything above from "good ideas" into requirements.

Every agent ends up with an owner sooner or later. The only question is whether you pick them before the incident — while they can still prevent it — or the incident picks them for you.

Quick tips

A few tips that are easy to implement and make a real difference:

1. Every model call goes through tracing. A gateway like Cloudflare AI Gateway or tools like Langfuse log every LLM call. When something happens — it will — the postmortem runs on data, not on memory.

2. Use the latest models. Running an old model is like running an unpatched library: newer models ship the most recent safety training — they resist injections better and follow instructions better. "If it works, don't touch it" works against you here.

3. Deterministic parameters. If a parameter defines whose data it is, the LLM doesn't decide it: it's injected from the outside, from the authenticated session. The model can choose which transaction to look up — never which user's. That's the defense against cross-user access: the model can't leak another user's data.

What to ask an AI agent vendor

If you're evaluating AI agents for your operation, these are the questions that separate a nice demo from a serious architecture:

A serious vendor won't answer "the prompt tells it not to". They'll show you:

• The permissions architecture.
• The guardrails running outside the model.
• The points where a human has the final word.

Non-deterministic AI is an extraordinary tool. We just have to stop asking it to also be its own security system.

---

Sources:

• The Guardian and Live Science — coverage of the PocketOS incident (April 2026)
• OpenAI — gpt-oss-safeguard, open-source models for policy-based guardrails