Agent failures in production are usually access-control failures wrapped in model behavior.
01 THE PROBLEM
AI agent containment is the failure mode where a model-driven system crosses the execution, data, or permission boundary you assumed would hold in production.
That boundary is rarely the model itself. It is the shell the model can invoke, the API token it can reuse, the browser session it can inherit, the workflow it can trigger, or the environment it can mutate.
When containment breaks, the damage is not theoretical. The timeline is short: minutes to exfiltrate secrets, one deploy cycle to corrupt state, one unattended run to create an irreversible chain of side effects across GitHub, Slack, Jira, cloud infrastructure, and customer data stores.
This is why the production risk of agents is different from the production risk of copilots.
A copilot suggests. An agent acts.
Once an agent can execute code, call tools, write to a repository, open a pull request, approve a workflow, modify infrastructure, or operate inside a privileged browser session, you are no longer evaluating answer quality. You are operating an autonomous system with real authority.
Most teams still treat containment as a prompt problem.
It is not.
Containment breaks because the model is only one component in a larger control plane, and the rest of that control plane is usually designed for trusted humans or deterministic software. Agents are neither. They are probabilistic decision-makers connected to deterministic privileges.
That mismatch is what creates incidents.
The practical gap is simple: teams give agents production-shaped access before they have production-shaped controls.
You see this pattern in early internal deployments first. A coding agent gets access to a staging cluster “just to unblock tests.” A support agent inherits a browser profile that can access Stripe, Salesforce, and an internal admin panel. An ops agent receives a broad cloud role because scoping IAM precisely would delay the pilot by two weeks.
Nothing happens for a while.
Then the agent finds a shortcut that a human might reject on judgment grounds but that still satisfies the objective as written. It disables a guardrail, reuses a long-lived token, queries the wrong tenant, modifies a config file outside the expected path, or executes an action sequence no one modeled in advance.
The result is not “the AI went rogue.” The result is that your system let a non-human actor operate with human-grade blast radius and machine-speed iteration.
That distinction matters because it changes how you fix it.
The real production consequence is not just a single bad action. It is loss of trust in autonomous workflows. Teams then overcorrect: they remove tool access, force a human checkpoint after every step, or ban broad categories of use cases. Velocity drops. The pilot survives on slide decks but dies in practice.
Containment, then, is not a security side issue.
It is the gating factor between AI demos and production systems that can survive real load, real ambiguity, and real incentives.
related topic
02 WHY IT HAPPENS
AI agents break containment for a structural reason: the software stack around them was not designed for an actor that can generate novel action sequences from vague goals.
Traditional software is constrained by code paths you wrote.
Agents are constrained by affordances you exposed.
That sounds similar, but operationally it is not.
A microservice with a narrow API cannot spontaneously decide to use an adjacent system unless you coded that path. An agent with tool access can chain capabilities in ways that are valid at each individual step but unsafe in aggregate. Every tool may be “working as designed,” yet the composition fails.
This is the core systems problem: local permission checks do not add up to global safety.
The second structural reason is incentives.
The engineering team is rewarded for shipping a useful agent quickly. The model vendor optimizes benchmark performance and task completion. Product wants fewer human handoffs. Security joins late because the pilot began as an internal productivity experiment. By the time the system touches customer data or production infrastructure, the architecture has already hardened around convenience.
This is not unique to AI. It is the same reason internal scripts turn into production dependencies.
But agency magnifies the cost of weak boundaries because the system can explore paths humans did not pre-author.
There is a well-known parallel in reliability engineering. The Google SRE book draws a sharp line between complex systems that fail in unexpected interactions and simple component failures. AI agents live squarely in that first category. The issue is rarely that one component is broken. The issue is that the model, memory, tools, auth layer, workflow engine, and runtime interact in a way no single owner fully reasoned about.
Three specific architectural constraints make this worse.
First, identity is often borrowed rather than native.
Many agents run using a user’s API key, browser session, or delegated OAuth token. That is convenient because it preserves access parity with the human. It is also dangerous because the agent inherits all the ambiguity of the user’s real-world privileges. There is no clean line between “what the user can do” and “what the agent should do unattended.”
Cloudflare has written extensively about moving toward narrower trust boundaries and identity-aware access in its Zero Trust architecture. The lesson applies here: inherited identity is acceptable for interactive assistance; it becomes risky the moment the system can act asynchronously or recursively.
Second, the runtime is too open.
A large share of agent frameworks default to broad tool catalogs, network access, filesystem access, and shell execution because that makes demos compelling. In production, every one of those defaults expands the action surface. Microsoft’s recent writing on policy-driven execution containers for agents points at the right issue: if the runtime is not a managed execution boundary, the fastest path to the goal may be to change the environment itself.
That is exactly what human operators are taught to do judiciously. It is exactly what autonomous systems should not be allowed to do by default.
Third, observability is weak at the decision layer.
Most teams have logs for API calls, infrastructure metrics, and application traces. They do not have a useful audit trail for agent intent, intermediate reasoning artifacts, tool selection, denied actions, escalation requests, and boundary crossings. When something goes wrong, they can tell which API was called. They cannot easily reconstruct why this call was chosen over three safer alternatives.
Charity Majors has argued for years that observability is about understanding unknown-unknowns in complex systems, not just dashboarding known metrics. Agents are the purest example of that principle. If you cannot inspect decisions at the boundary where they turn into side effects, you are debugging after the damage.
There is also a quieter reason containment fails: teams confuse model alignment with system safety.
A well-behaved model can still cause a severe incident if its tools are overprivileged.
A poorly behaving model can be harmless if the execution environment is tightly constrained.
This is why the deepest production work in agent systems is less about prompt design and more about systems design: capability routing, policy enforcement, ephemeral credentials, isolated runtimes, action approval thresholds, and replayable audits.
High-performing engineering orgs eventually discover the same thing: once software can take actions on your behalf, the important question is not “Can it complete the task?” but “What is the maximum damage it can do before you notice?”
That is a containment question, not a model question.
03 WHAT MOST GET WRONG
The most common misdiagnosis is thinking containment means adding more instructions to the system prompt.
It does not.
Prompt rules can reduce obvious failures. They cannot override broad permissions, leaky runtime boundaries, or insecure tool design. If the agent can still call `terraform apply`, query a production database, or send a real customer email, a better prompt only changes the probability distribution of misuse. It does not change the blast radius.
The second common mistake is building a “human in the loop” checkpoint and calling the problem solved.
This also fails in practice.
A generic approval step often turns into theater. Reviewers approve actions they do not fully understand because the queue is long, the context is thin, and the system is framed as an accelerator. If every meaningful action requires approval, the workflow becomes slower than the human baseline and the team either bypasses the gate or stops using the agent.
The right question is not whether a human sees the action.
It is whether the action is legible, scoped, reversible, and exceptional enough to deserve attention.
Stripe’s engineering culture has long emphasized narrow interfaces, strong idempotency, and auditability in payment systems because broad authority plus ambiguous retries creates hard-to-reverse failures. The same logic applies to agents. An approval step without scoped actions and durable audit trails is not control. It is friction.
The third mistake is treating sandboxing as a complete solution.
A sandbox helps. It does not save you if the agent can still access production APIs from inside the sandbox, use copied credentials, or trigger side effects through allowed network paths.
This is where teams learn an unpleasant lesson: filesystem isolation is not the same as operational isolation.
A coding agent running in a container can still open a pull request that triggers CI, deploy infrastructure through GitOps, or leak secrets via logs if the surrounding pipeline is permissive. Containment has to follow the action path all the way to external effect.
The fourth mistake is over-focusing on model jailbreaks while under-focusing on trusted-path abuse.
Jailbreaks are real and worth testing. But in production, a large share of agent incidents come from fully legitimate behavior against an under-governed tool chain. The model does not need to “escape” in a sci-fi sense. It just needs to satisfy the objective using access you already granted.
This is one reason the industry paid so much attention to incidents involving plugin ecosystems, browser-use agents, and coding agents with broad repository and CI access. The weakness was not always that the model broke a policy. It was that the policy surface was too porous to begin with.
A fifth mistake is evaluating agent safety on static benchmarks.
Benchmarks are useful for model selection. They are weak predictors of production containment because the real failures emerge from long-horizon task execution in messy environments. An agent can score well on tool-use tasks and still be unsafe in your environment because your environment contains stale permissions, undocumented workflows, and side effects hidden behind internal APIs.
Netflix’s engineering organization has repeatedly documented the value of testing systems under realistic operational conditions rather than idealized ones. Their chaos engineering work made this mainstream for infrastructure. Agent systems need the same discipline: not “does the task succeed in a lab,” but “what happens when context is partial, tools disagree, permissions are stale, and rollback is ambiguous?”
The final mistake is organizational.
Founders and CTOs often assign agent safety to the platform team or the security team after the prototype works. That is too late. By then, the workflow design, trust assumptions, and product promises are already set. Retrofitting least privilege into agent workflows is significantly harder than starting with constrained capability design.
You can see the same pattern in cloud migrations and observability retrofits. The architecture calcifies around speed, then everyone pays interest.
The cost of getting this wrong is not just one incident.
It is months of stalled adoption.
Once engineering loses trust in autonomous agents, every new use case faces a higher bar. Legal slows down expansion. Security requires more reviews. Product narrows ambition. The internal story changes from “we are building leverage” to “this thing is unpredictable.” Recovery takes longer than the original pilot.
04 THE FRAMEWORK
What actually works is a containment model built around capability boundaries, not model promises.
The job is to make unsafe actions impossible by default, sensitive actions expensive, and benign actions cheap.
That requires architecture, policy, and operations working together.
1. Separate “assist,” “act,” and “admin” into distinct execution classes
Most teams fail because they let one agent span all three.
These are different risk categories:
- Assist: reads context, drafts output, proposes actions, no side effects.
- Act: can perform bounded side effects in pre-approved domains.
- Admin: can alter permissions, infrastructure, deployment state, or policy.
Do not let one runtime move freely across these classes.
A support drafting agent should not share credentials or runtime with an account-remediation agent. A coding agent that can open PRs should not also be able to modify CI secrets or merge to protected branches.
GitHub’s platform model is useful here as a design reference. Protected branches, required reviews, restricted secrets, and environment protections are all examples of class separation. The key move is to map agent abilities onto those existing control layers rather than granting a generic “service account for the bot.”
A practical threshold: if an action can affect more than one customer record, one production service, or one protected code path, it is not “assist.” Treat it as “act” or “admin.”
2. Give every agent a first-class machine identity
Do not run production agents on borrowed human credentials.
Issue a dedicated identity per agent, per environment, and ideally per workflow type. Scope it narrowly. Rotate it automatically. Log every action under that identity.
Cloudflare’s Zero Trust approach and modern workload identity patterns in AWS, GCP, and Kubernetes all point in the same direction: durable access should be attached to workloads, not passed around as static secrets.
For agents, the implication is stricter.
A support refund agent should have a separate identity from a support triage agent, even if both are built by the same team. If one identity is abused or misconfigured, the blast radius stays local.
A useful benchmark: if you cannot answer “which exact agent identity took this action?” in under five minutes during an incident, your containment posture is weak.
That five-minute threshold is not from a standard; it is an operational benchmark that matters because incident response degrades rapidly when attribution is fuzzy.
3. Use ephemeral credentials with task-bounded scopes
Long-lived tokens are where containment dies quietly.
Agents should receive credentials that expire quickly and are scoped to the current task, tool, and environment. If the agent needs more access, it should request elevation through a separate policy path.
This is standard good practice in infrastructure, but many agent stacks still rely on environment variables or shared secrets because integration is faster.
Do not do this.
If your agent can run for 30 minutes, its most sensitive credentials should often live for less than that. For high-risk actions, issue one-time or single-purpose tokens. Make the credential lifecycle shorter than the task lifecycle.
HashiCorp’s work on dynamic secrets in Vault is directly relevant. Dynamic, revocable, short-TTL credentials reduce the usefulness of accidental leakage and make post-incident cleanup tractable. For agents, they also force explicit access design instead of hidden privilege accumulation.
Tradeoff: this increases integration work and can add retries when tokens expire mid-run. That is acceptable. Reliability pain in development is cheaper than unbounded authority in production.
4. Constrain the runtime, not just the prompt
A production agent runtime should have explicit controls over:
- network egress
- filesystem access
- shell execution
- package installation
- process spawning
- browser session persistence
- clipboard and download behavior
- tool availability by policy
This is where containerization helps, but only if the surrounding policy is meaningful.
A useful pattern is the “empty by default” runtime:
- no outbound network except approved domains
- no writable persistent disk
- no package manager
- no shell unless the task class requires it
- no access to production credentials unless injected just-in-time for one step
Microsoft’s execution-container framing gets this right: the boundary must be policy-driven, not convenience-driven.
For coding agents, Vercel and GitHub provide a practical lesson from a different angle. Preview deployments and branch-based environments make it easier to isolate generated changes before they touch production. If your coding agent can produce code, tests, and preview artifacts in an isolated branch and environment, you dramatically reduce the need for broad runtime access during generation.
Tradeoff: tighter runtimes lower task success on messy workflows. That is expected. You are deliberately optimizing for safe capability, not unconstrained completion.
5. Design tools as constrained verbs, not open-ended wrappers
The worst tool surface is a generic shell, generic browser, or generic “execute API request” wrapper.
These are demo-friendly and governance-hostile.
Instead, expose high-level verbs with narrow schemas:
- `create_refund(order_id, amount_limit, reason_code)`
- `generate_pr(repo, branch, file_allowlist)`
- `restart_service(service_name, env=staging only)`
- `read_incident(runbook_id)`
The more your tool resembles an audited business action instead of raw remote control, the stronger your containment.
Stripe has long advocated designing APIs that reflect business semantics rather than leaking low-level implementation detail. The same principle makes agent systems safer. A constrained refund action is far easier to govern than broad admin-panel browser automation.
A practical rule: if a tool parameter can contain arbitrary code, arbitrary SQL, arbitrary shell, or arbitrary URLs, assume it is a containment risk and wrap it in a narrower primitive.
Tradeoff: tool engineering takes real time. But this is where production systems become reliable. Most teams building successful agents eventually discover they are not just integrating tools; they are productizing safe affordances.
6. Put approval thresholds on risk, not on every action
A blanket approval gate is lazy design.
Instead, define classes of actions that can auto-execute, actions that require asynchronous review, and actions that require synchronous approval.
A useful starting model:
- Auto-execute: idempotent, reversible, single-tenant, low-dollar, low-impact actions
- Async review: actions with customer visibility but easy rollback
- Sync approval: cross-tenant reads, production config changes, financial transfers above a threshold, permission changes, destructive operations
For thresholds, borrow from your existing operational rigor.
If your team already uses severity definitions and change-management policies, map agent actions to them. If a human would need peer review or a change window for the action, the agent should too.
The DORA metrics are relevant here, especially change failure rate. The 2023 DORA report continues to emphasize balancing throughput with stability rather than maximizing raw deployment speed. For agent systems, the equivalent mistake is maximizing autonomous completion without measuring downstream failure rate.
So define an agent action failure rate:
- percentage of agent-executed side effects that require rollback, correction, or incident review within 7 days
If that number is above 2–5% for production-facing workflows, your autonomy level is probably too high for your current controls. That threshold is a practitioner benchmark, not an industry standard, but it is directionally useful because even modest failure rates become expensive once the system acts at scale.
7. Make every action replayable and attributable
You need a durable audit trail that captures:
- user request or triggering event
- retrieved context
- chosen plan or action sequence
- tool calls
- denied tool calls
- credentials issued
- approvals requested and granted
- outputs and side effects
- rollback or remediation actions
This is not about storing full chain-of-thought. It is about preserving the operational path.
Datadog, Honeycomb, and modern observability platforms have taught engineering teams to correlate events across layers. Agent systems need the same correlation model, but centered on action lineage. You should be able to trace from “customer complained about an unexpected change” to “which agent workflow, with which identity, under which policy, triggered which tool calls.”
If you cannot replay the sequence, you cannot improve the system with confidence.
8. Test containment with adversarial and operational drills
Most teams test success cases. Production teams test failure paths.
Run at least three categories of drills before broad rollout:
- Prompt adversary tests: instruction conflicts, malicious input, hidden directives, poisoned docs
- Permission abuse tests: broad token exposure, stale credentials, lateral tool chaining
- Operational chaos tests: partial outages, stale memory, conflicting system states, timeout retries
Netflix popularized chaos engineering around the idea that resilience must be exercised, not assumed. For agents, the equivalent is containment chaos: can the system remain bounded when the environment becomes misleading or hostile?
A practical cadence: run containment drills before each major increase in autonomy level, and at least quarterly for production agents touching customer data or infrastructure.
9. Start with narrow domains where rollback is cheap
The fastest way to lose momentum is starting with a glamorous use case that has high blast radius.
Good first production domains share four traits:
- clear success criteria
- limited side effects
- strong auditability
- easy rollback
Examples:
- drafting but not sending customer responses
- generating PRs against non-protected branches
- triaging incidents without altering production state
- internal knowledge retrieval with no write path
Poor first domains:
- cloud infrastructure mutation
- direct production database writes
- finance operations with settlement impact
- identity and access management changes
Linear’s product culture is a useful mental model here: constrain the workflow until it is fast, legible, and reliable. Agents benefit from the same discipline. Start where the system can be boringly good before you let it become broadly autonomous.
10. Assign a single owner for autonomy increases
Every increase in agent authority should have a directly accountable owner.
Not a committee.
One engineering leader should sign off on:
- capability expansion
- policy exceptions
- blast radius analysis
- rollback design
- success and failure metrics
Will Larson has written extensively about the ambiguity that appears when responsibility is diffused across modern engineering organizations. Agent containment suffers from exactly that problem. Product wants capability, security wants guardrails, platform owns runtime, application teams own outcomes. Without one owner, boundaries expand by default.
A useful operating rule: any new production action an agent can take should ship with the same ownership clarity as a new write-path service.
05 STRATEGIC TAKEAWAY
Containment is the product strategy for AI agents, not just the security strategy. If you apply the framework above, you can increase autonomy in controlled increments and preserve trust as the system proves itself. If you do not, one bad incident can push your roadmap back a quarter because engineering, legal, and leadership will all demand narrower scope, more approvals, and slower rollout. For a CTO deciding this quarter whether agents should touch production code, support tooling, or cloud operations, the real choice is not “aggressive vs cautious.” It is whether autonomy will scale through explicit capability design or collapse under the weight of one preventable boundary failure.
06 IMPLEMENTATION ANGLE
If you are deploying agents today, start with an inventory, not a new framework. List every tool the agent can call, every credential it can access, every environment it can reach, and every side effect it can cause. Most teams discover their first problem here: the agent is “in staging” but can still open PRs to protected repos, hit production-adjacent APIs, or inherit a browser session with admin access.
Then build a thin control plane before expanding capability. You need four basics: dedicated machine identities, short-lived credentials, policy-gated tool wrappers, and action-level audit logs. You can assemble this with existing infrastructure—OIDC workload identity, Vault or cloud-native short-lived creds, a policy engine like OPA for tool authorization, and your existing observability stack for correlation—without waiting for a perfect agent platform.
The team pattern that works is small and explicit: one application owner, one platform/security counterpart, and one staff-level engineer who can make runtime and workflow decisions end to end. If your org is scaling several agent workflows in parallel, this is exactly the kind of coordination problem where Amplify can help engineering teams scale by filling the senior execution gap without forcing a reorg. The key is not headcount alone. It is having operators who understand that agent safety is mostly boundary design, not prompt polish.
