Blog

A Prompt Is Not an Authorization Check

A practical way to decide which security controls an AI agent actually needs.

· Agentic AI, Security

An AI model hands a proposed action to a trusted checklist inside a boundary before it reaches a database, while a dashed route to the outside internet is blocked.

There’s a lot of pressure to ship AI applications quickly. A working agent can be put together in days, and security often becomes something to revisit after the demo.

That isn’t always unreasonable. An internal experiment with disposable data and a tightly restricted set of tools is very different from an agent that can modify customer records or send confidential information outside the company.

The problem is carrying the prototype’s assumptions into production without reconsidering what the agent is allowed to do.

Start with its authority

Before debating prompt-injection defenses, I ask three questions:

  1. What can the agent read? Which documents, customer records and other confidential resources can it reach?
  2. What can it do? Can it write to databases, send messages, execute code or trigger irreversible operations?
  3. Where can information go? Do its output or its tools communicate with external services?

The answers say much more about the actual risk than whether we call something an “agent”. A system with no sensitive data and no consequential actions can often stay simple. A system with broad privileges needs controls that sit outside the model.

Diagram of a booking assistant and three dashed boundaries. Reads: policy docs, customer records, untrusted web pages and email. Does: booking API, notifications, customer record writes. Depends on: model provider, MCP tools, frameworks and models. An evidence bar below lists threat model, tests, scoped logs and incident response.

The same questions drawn as boundaries for a booking assistant: what it reads, what it does, and (a third one I use in practice) what it depends on. From my talk Architecting Production AI Systems for Security and Stakeholder Trust.

The model proposes; the backend decides

Imagine a reservation assistant. The model interprets “Move my booking to Thursday afternoon” and proposes a structured action. Before anything changes, a trusted service checks:

  • Is the user authenticated, and are they allowed to modify this reservation?
  • Is the operation permitted, and is the new slot still available?
  • Has the operation already been completed? Would a retry duplicate it?
  • Does this particular operation require the user’s confirmation?

Putting “only modify the user’s own bookings” in the system prompt is sensible, but it isn’t an authorization mechanism. The model should never be the authority on whether its own proposal is allowed.

The LLM turns a customer request into a typed proposal carrying customer_id c_4711. A trusted execution service checks it: schema validation and session identity pass, authorization and business rules fail. Bookings shows 0 writes and the audit log records the denial.

If an attacker fully controls the proposal, the damage is still bounded. A forged customer_id passes the schema check and is rejected at authorization, leaving zero writes and an audit entry. From my talk Architecting Production AI Systems for Security and Stakeholder Trust.

Correct access control can still leak

Suppose a user is entitled to read an internal document, and the retrieval layer correctly hands it to the assistant. Now a piece of attacker-controlled text in the same context instructs the model to embed part of that document in a URL pointing outside the company.

The read was authorized. The use of the information was not.

Triangle of private data, untrusted content and an outbound channel, with exposure at the centre. A flow policy in code blocks customer records going to collector.example and lets a confirmation through to the logged-in customer.

Simon Willison’s “lethal trifecta”: private data, untrusted content and an outbound channel in one system. A flow policy enforced in code decides what may leave. From my talk Architecting Production AI Systems for Security and Stakeholder Trust.

This is why safe output rendering and restrictions on outbound destinations matter. A client that automatically fetches any image URL the model generates has given the model an outbound channel. The EchoLeak disclosure is a useful case study. A crafted email sitting in a Microsoft 365 mailbox was enough to steer Copilot into leaking internal data through generated Markdown, without the user clicking anything. Microsoft fixed it server-side as CVE-2025-32711.

Four steps: a crafted email arrives, an ordinary question pulls it into context, the model writes a URL with sensitive data encoded in it, and the client fetches it as an auto-loaded image. Tags: demonstrated by researchers, patched, not known to be exploited in the wild.

The EchoLeak chain. No step needed the user to click: the auto-loaded image was the outbound request. From my talk Architecting Production AI Systems for Security and Stakeholder Trust.

What I’d actually test

For a production agent with sensitive access, I want evidence that:

  • A user cannot retrieve another user’s documents, including through caches and alternative retrieval paths.
  • A model-generated tool call cannot bypass backend authorization or business rules.
  • Disallowed outbound destinations are blocked, even when they appear inside plausible-looking content.
  • Retries cannot silently repeat consequential operations.
  • Model or tool updates trigger regression tests for the behaviors we rely on.

Few of these are LLM-specific problems. They’re ordinary software engineering controls, applied to a component that makes less predictable decisions.

My rule of thumb: keep low-risk experiments lightweight, but enforce the important restrictions where the model can’t talk its way around them.