Policy, not prompts

The moment an agent is allowed to change something, the instruction in its prompt stops being a boundary and becomes a suggestion. Prompts are text the model interprets; policy is a rule the system enforces before the action reaches the world. Every serious question about letting agents act is really a question about which of those two you built.

Why “you must never” is not a control

A system prompt saying the agent must never issue a refund above a threshold is not a control, for three reasons that have nothing to do with the model being badly behaved.

It can be misread — natural language is ambiguous and instructions compete with each other under pressure. It can be displaced — enough context, a long conversation or a convincingly-worded document can push the constraint out of the model’s effective attention. And it leaves no evidence: when something goes wrong, there is no record separate from the conversation that shows what the boundary was at that moment.

None of this makes agents unusable. It makes the prompt the wrong place for the boundary.

What a policy layer has to express

A policy engine worth the name answers five questions for every action an agent might take, and answers them outside the model.

Who is acting. Not “the assistant”, but which agent, on whose behalf, with which delegated authority. Identity is the thing everything else hangs on.

What class of action this is. Reading is not writing. Drafting is not sending. Preparing a payment is not making one. Most useful agent work sits in the safe end of that range, and collapsing the distinction is how teams end up too frightened to deploy anything.

Within which limits. Amounts, counts, time windows, which customers, which accounts, which shelves. Limits are boring and they are the whole game.

With what evidence. What the agent must have read, checked or reconciled before the action is permitted. This is where a policy engine quietly improves quality rather than just restricting it.

Requiring whose confirmation. Nothing, a named human, two humans, or a human plus a cooling-off period. A gradient, not a switch.

The gradient is the point

The debate about agent autonomy is usually framed as trust: do you trust it or not. That framing produces bad systems, because it forces one decision across wildly different actions.

A far better model is earned scope, per process. An agent starts read-only. It proposes, a human disposes, and both the proposal and the outcome are recorded. When the record shows it has been right about that specific process for long enough, the boundary moves — for that process only, with a limit attached, and with the ability to move it back.

That is not a philosophical position on machine intelligence. It is how you would onboard a capable new employee, and for the same reasons.

Audit is a by-product, not a project

Here is the argument that usually wins the budget. If policy lives in a prompt, then answering “why did the system do that in March” means reconstructing a conversation, if you still have it, and inferring intent from text.

If policy is a first-class object, the answer is a query: this policy version, these inputs, this evaluation, this outcome, this human confirmation. The evidence a regulator or an auditor will ask for is produced as a side effect of running the system correctly, not assembled in a panic the quarter before an inspection.

The uncomfortable implication for product design

A policy engine cannot be bolted on. It has to sit underneath the thing taking the action, which means it belongs in the operating system rather than in each application — otherwise you get eleven permission models that disagree, which is the situation we have today, only now with agents in it.

That is the argument for putting agents in an OS rather than in a product. Not because it sounds more impressive. Because a boundary is only a boundary if there is exactly one place it can be enforced.