Agentic AI turns cheap prompts into expensive systems by multiplying tokens, latency, and operational risk.
01 THE PROBLEM
Agentic AI is the failure mode where a team prices a single model call and deploys a multi-step software system.
That gap is where budgets get wrecked.
A chatbot answers once. An agent plans, calls tools, retries, critiques itself, pulls context, writes intermediate state, and often loops until a stopping condition is met. The visible prompt is one line item. The actual cost is a chain reaction across model inference, retrieval, orchestration, databases, observability, queues, human review, and incident response.
The consequence shows up fast. A prototype that looks affordable in week one becomes a line item the CTO has to explain by the end of the quarter. In practice, the first surprise is not usually GPU capacity. It is runaway per-task cost, unpredictable latency, and no clean way to attribute where the spend came from.
That is why “cost per token” is too small a unit of analysis.
InfoWorld’s coverage of agentic AI cost lays out the arithmetic clearly: an agent consuming 2 million tokens per day uses 730 million tokens per year. The number matters not because 2 million tokens sounds large, but because most teams reach it accidentally. They add one more retrieval pass, one more verifier model, one more tool call, and one more retry policy. The system crosses from “feature” to “workflow engine” without anyone redoing the economics.
The Stanford Digital Economy Lab highlighted the more uncomfortable point: agentic tasks can be uniquely expensive, with token consumption reaching 1000x more than code reasoning and code chat in certain task categories. Even if that figure reflects the upper end of agentic behavior rather than a universal average, it captures the core operating truth: once the model is allowed to think, branch, and act, token usage stops being linear.
This matters more for engineering leaders than for procurement.
A procurement team can negotiate model discounts. A CTO has to answer harder questions:
- What is the true marginal cost of each successful task?
- How much of that cost comes from retries and failed tool paths?
- What latency budget gets consumed by orchestration rather than inference?
- Which customers or workflows are economically viable at current autonomy levels?
- What reliability target is realistic when your “application” depends on an LLM, a vector store, half a dozen APIs, and a planner that may change behavior model to model?
If you do not answer those questions early, agentic AI creates a familiar but more volatile version of cloud overspend. The difference is that cloud waste is usually visible in infrastructure graphs. Agent waste is often hidden inside “successful” requests.
The most dangerous version of the problem is that teams mistake activity for value.
An agent can generate impressive traces, dozens of tool calls, and long chains of reasoning while producing no better business outcome than a deterministic workflow plus one classification step. By the time finance asks for unit economics, engineering has already coupled the product experience to an expensive architecture.
That is the real cost of agentic AI: not just more tokens, but systems whose economics degrade faster than their usefulness improves.
02 WHY IT HAPPENS
The structural reason is simple: autonomy multiplies variance.
A standard software request has a bounded path. A checkout flow, search request, or webhook handler may be complex, but engineers know the likely execution graph. Agentic AI breaks that predictability. It introduces conditional execution paths driven by a probabilistic model. Every path expansion increases spend uncertainty.
There are four root causes.
1. The control loop is invisible at prototype time. Early demos hide the cost structure because the happy path dominates. One prompt, one answer, one user, one notebook. The planner looks clever. The retries are few. The context window is manageable.Production is different.
Real workloads include malformed inputs, stale APIs, tool failures, permission issues, partial context retrieval, and users who ask for ten things in one request. The agent responds by generating more tokens and more calls. That means higher average cost, but the deeper problem is a much fatter tail. Your P95 and P99 spend per task can become economically absurd long before the average looks alarming.
This is the same systems lesson Stripe has written about in a different domain: production complexity emerges from edge cases, retries, and failure handling, not from the clean path in design docs. Agentic systems amplify that principle because the software is deciding how much software to run.
2. Teams optimize for task completion before they optimize for bounded execution. This is understandable. The product milestone is usually “make the agent work.” The engineering milestone should be “make the agent stop.”Those are not the same thing.
A capable planner, given enough retries and tool access, can often improve completion rate. It can also quietly double or triple the cost to achieve marginal gains. Teams celebrate benchmark wins while ignoring execution envelopes.
Google’s SRE book is useful here even though it predates LLM agents. Its core discipline is that reliability requires explicit budgets and error boundaries. Agentic AI needs the same treatment for cost and latency. If a workflow has no hard cap on steps, tokens, or tool calls, it is not an engineered system. It is an open tab.
3. Vendors price inference cleanly while the rest of the stack prices itself indirectly. Token prices are legible. The rest of the architecture is not.You can calculate input and output token cost from a pricing page. It is harder to capture the second-order costs:
- retrieval infrastructure
- embedding refresh jobs
- orchestration runtimes
- queue depth during spikes
- audit logs for regulated actions
- evaluation pipelines
- fallback models
- human review queues
- vendor lock-in from tool abstractions
- engineering time spent tracing failures no APM was built to explain
EY has made this point directly in its analysis of agentic AI costs: tokens are only one part of the total cost, which also includes infrastructure, governance, risk, and operating model changes. That statement is less interesting as consulting advice than as an engineering accounting principle. If your spreadsheet only includes model usage, it is wrong by design.
4. Most teams still lack the observability primitives for model-driven workflows. Cloudflare, Datadog, and PostHog have all pushed toward better observability for modern application paths, but most internal AI stacks still cannot answer basic questions such as:- Which tool caused the retry cascade?
- Which customer segment generates the highest cost-to-success ratio?
- Which prompts are triggering context bloat?
- Which model version increased average steps per completed task?
- What percentage of spend came from tasks that should have been terminated early?
This is partly a tooling gap and partly an architectural one. Traditional traces assume deterministic spans. Agent traces need cost attribution, branch accounting, and semantic state transitions. Without that, teams only discover problems in aggregate monthly bills or anecdotal support escalations.
The result is an incentive misalignment.
Product wants more autonomous behavior because autonomous demos are compelling. Engineering wants reliability. Finance wants predictability. Sales wants broad claims. Nobody owns the cross-functional metric that matters most: cost per successful action within latency and quality thresholds.
That metric should sit next to uptime and conversion in every serious AI product review.
Company examples outside AI help reinforce the lesson. Netflix’s engineering culture has long emphasized measured experimentation and operational visibility because distributed systems fail in non-obvious ways. Cloudflare consistently writes about making cost, performance, and security first-class architectural constraints, not afterthoughts. Linear’s product organization is notable precisely because it resists accidental complexity; that discipline matters even more in AI systems where complexity self-generates.
Agentic AI is expensive for the same reason microservices became expensive when teams adopted them casually: every additional degree of freedom creates more states to manage. The difference is that an LLM can generate those states faster than your architecture review process can constrain them.
03 WHAT MOST GET WRONG
The common misdiagnosis is that agentic AI is expensive because frontier models are expensive.
That is not the main problem.
The bigger issue is unbounded execution wrapped in weak product discipline. Teams pick a powerful model, allow broad tool use, stream every intermediate step into the context window, and then blame inference pricing when costs spike.
Switching to a cheaper model does not fix that architecture. It just lets the system fail more cheaply.
The second mistake is treating token volume as the primary optimization target.
It is not.
The right target is cost per successful, policy-compliant outcome at an acceptable latency. Token reduction can help, but it is a means, not the metric. A workflow that uses 40% fewer tokens but doubles failure review by humans is not cheaper. A workflow that completes 15% more tasks but takes 5x longer may still destroy user trust.
This is where many teams repeat an old cloud-era pattern: they optimize the visible bill instead of the business system.
A third mistake is assuming more autonomy is always better product design.
It often is not.
The most effective production AI systems are frequently narrow, staged, and constrained. GitHub Copilot succeeded because the core loop was tightly scoped around code suggestion, not because it acted as a free-roaming software agent from day one. When GitHub expanded toward agentic coding workflows, it did so on top of an already disciplined product surface and strong telemetry. That sequencing matters.
The fourth mistake is failing to separate “expensive because valuable” from “expensive because confused.”
A tax research agent that saves a lawyer two billable hours may justify high per-task cost. An internal meeting summarizer that burns the same budget does not. Teams collapse these categories because they lack a value-weighted routing policy. Every task gets the same broad agent treatment.
That is architectural malpractice.
The pattern has shown up before in adjacent systems. Charity Majors has argued for years that teams often drown in observability cost because they instrument everything without defining what decisions the data needs to support. Agentic AI creates the same trap in reverse: teams execute everything and only later ask whether the execution graph was necessary.
One concrete failure pattern appears in retrieval-heavy architectures. Teams assume that adding more context improves accuracy. In practice, it often degrades both cost and performance. Larger prompts increase spend, increase latency, and can reduce answer quality when relevant signal is diluted. This is not an abstract concern. Across production search and ranking systems, companies like Airbnb and Shopify have repeatedly emphasized careful ranking, filtering, and relevance over brute-force inclusion. The same discipline applies here: better context selection beats bigger context windows.
Another common failure is overbuilding agents where deterministic software should dominate.
If a workflow is fundamentally stateful, policy-bound, and repeatable, use software first and models second.
A refund workflow is a good example. If policy thresholds, fraud checks, and escalation paths are known, the agent should classify intent, summarize context, and draft action rationale. It should not be the primary execution engine deciding from scratch what “refund policy” means every time. Stripe’s documentation and engineering writing consistently reflect this philosophy in payments: critical flows are bounded, observable, and policy-driven because financial software cannot rely on improvisation.
The final mistake is organizational.
Most companies assign agentic AI to a feature team without assigning ownership for unit economics. The result is predictable. The PM tracks adoption. Engineering tracks incidents. Finance tracks spend. Nobody tracks whether the product should keep doing the thing at all.
That gap is why teams can spend six months optimizing prompts on a product whose economics never worked.
04 THE FRAMEWORK
What actually works is not “use smaller models” or “prompt better.” It is a cost architecture.
You need to design agentic AI the way high-performing teams design any production system: with budgets, control points, routing logic, and explicit reliability boundaries.
Here is the framework.
1. Define the unit of value before you define the agent
Start with the business action, not the model capability.
A good unit is something like:
- cost per resolved support ticket
- cost per accepted code change
- cost per successful sales research brief
- cost per onboarded customer record
- cost per escalated fraud case correctly prioritized
A bad unit is:
- cost per conversation
- cost per session
- cost per thousand requests
Those are workload counters, not outcomes.
The right denominator forces hard choices. If your sales research agent costs $4 per account brief and increases AE productivity by 20 minutes per account, that might be viable. If your support triage agent costs $1.20 per case but only deflects tickets worth $0.40 in human time, the product is upside down no matter how elegant the prompt chain is.
This step sounds obvious, but it is where most projects fail. Teams define success as “agent completed task” rather than “business created more value than it consumed.”
2. Put hard execution budgets in code
Every agentic workflow needs ceilings, not aspirations.
Set explicit limits for:
- max total tokens per task
- max planner iterations
- max tool calls
- max retrieval chunks
- max wall-clock duration
- max retries per dependency
- max spend per completed action
If a task crosses the budget, terminate, degrade, or escalate.
The Google SRE model is relevant here: budgets create reliability by forcing tradeoffs. In agentic systems, they create economic sanity. A task that is allowed to exceed its intended cost envelope in pursuit of completion is not “resilient.” It is leaking money.
Practical thresholds depend on margin and workflow value, but the pattern is universal. For internal copilots, many teams benefit from an aggressive first budget: one planner pass, one retrieval stage, one execution stage, one verifier, then stop. If the system cannot produce value inside that envelope, broadening the loop should be a deliberate decision with measured ROI.
3. Route by task class, not by user entry point
Not every request deserves an agent.
Create a routing layer that classifies tasks into at least four buckets:
- deterministic software only
- single LLM call with tools disabled
- constrained workflow with fixed tools
- full agentic loop with monitoring and escalation
This is the architectural decision that saves money fastest.
A user may type everything into the same text box, but your backend should not treat every query the same. “Summarize this document” is not “investigate discrepancies across five systems and propose next actions.” One should be a cheap bounded call. The other may justify a more expensive loop.
Cloudflare’s product architecture often reflects this principle in another form: use the edge for what the edge is good at, route deeper only when necessary. The same logic applies to agentic AI. Cheap path first. Expensive path only when confidence, complexity, and value justify it.
4. Make context retrieval earn its keep
Retrieval is often the hidden tax in AI systems.
Most teams over-retrieve, under-rank, and then pay twice: once in infrastructure and once again in prompt inflation.
Use a retrieval policy with measurable gates:
- top-k defaults should be low, often 3–8 rather than 20+
- chunk size should reflect answer granularity, not ingestion convenience
- retrieval quality should be evaluated against answer lift, not recall alone
- stale documents should age out automatically
- every chunk added to context should be attributable in traces
This is where Figma- and Shopify-style product discipline matters, even if the exact blog post is not about LLMs. Both companies are known for reducing product friction by keeping interfaces and systems focused. Retrieval layers need that same restraint.
If five chunks produce the same answer quality as fifteen, the extra ten are operational waste.
5. Separate planning, acting, and verifying models
Do not pay frontier rates for every subtask.
A common production pattern is:
- use a stronger model for plan generation on ambiguous tasks
- use a cheaper model or deterministic code for routine transformations
- use specialized models or rules for verification
- reserve expensive fallback only for low-confidence outcomes
This architecture usually beats the “one premium model does everything” approach on both cost and control.
Vercel’s AI application guidance has consistently pushed developers toward composable patterns rather than monolithic model usage. That is the right instinct here. Different stages have different requirements. Planning needs reasoning. Extraction needs consistency. Verification needs policy precision. Treating them as one inference class is lazy systems design.
The tradeoff is complexity. Multi-model routing increases test surface area and vendor management burden. If your team is under ten engineers and your workflow volume is low, the simpler architecture may still be the right short-term call. But if usage is growing, this split becomes one of the highest-leverage controls you have.
6. Instrument cost per step, not just per request
You need traces that answer financial questions.
For every workflow, log:
- tokens in and out by step
- tool invocations by type
- retrieval count and payload size
- retries and causes
- latency by span
- outcome class: success, degraded success, human escalation, terminated
- estimated dollar cost per span and per completed task
Datadog, PostHog, and newer LLM observability tools can help, but the key is conceptual, not vendor-specific. If your dashboard says “this endpoint cost $8,000 yesterday,” that is too late and too vague. You need “verification retries on CRM sync consumed 31% of yesterday’s spend and produced only 4% of successful completions.”
That level of attribution changes decisions.
It tells you whether to optimize a prompt, rewrite a tool, narrow permissions, or remove the feature.
7. Design human escalation as a product path, not an exception
Human-in-the-loop is not a failure of agentic AI. It is often the only thing making the unit economics work.
The mistake is escalating too late.
If an agent spends 90 seconds, six tools, and thousands of tokens before handing the task to a human, you have paid twice: once for the model loop and once for the reviewer. Better systems escalate early when uncertainty crosses a threshold.
This is where operational experience matters. Stripe and Airbnb both operate domains where support, trust, and risk workflows need clear handoffs and auditability. Agentic systems in similar domains should do the same. The best design is often not “more autonomy.” It is “earlier confidence scoring, better summaries, cleaner handoff.”
A good escalation packet includes:
- detected intent
- sources consulted
- actions attempted
- policy conflicts found
- concise rationale for escalation
That turns model spend into reviewer leverage rather than waste.
8. Benchmark against latency and reliability, not demo quality
DORA’s four key metrics are about software delivery performance, not AI directly, but the underlying principle holds: optimize systems against operational outcomes, not anecdotes.
For agentic AI, your scorecard should include at least:
- median and P95 cost per successful task
- median and P95 latency
- first-pass completion rate
- escalation rate
- policy violation rate
- retry-induced spend as a share of total cost
- percent of tasks completed within budget envelope
If your P95 cost is 5x your median, you likely have an execution-boundary problem. If your completion rate improves only by pushing P95 latency beyond what users tolerate, you do not have a product. You have a benchmark artifact.
The “5 to 30 times more tokens per task” figure cited in Gartner reporting and discussed by Cockroach Labs and Spheron is directionally useful here. Treat it as a planning assumption for agentic workloads until your own telemetry proves otherwise. If your financial model assumes only a 1.5x multiplier over chatbot usage, you are budgeting fantasy.
9. Use deterministic software for the skeleton
The agent should fill gaps, not provide the entire structure.
A durable pattern is:
- workflows, permissions, policy rules, retries, and side-effect execution live in code
- models handle interpretation, summarization, ranking, extraction, and draft generation
- critical actions require explicit checks before commit
This is how you avoid “prompt-driven architecture,” where business logic migrates into text blobs nobody can test properly.
HashiCorp’s engineering culture, shaped by infrastructure software, has long favored explicit state and predictable execution. That mindset is useful here. If your agent is mutating state in production systems, then state machines, idempotency, and permission boundaries matter more, not less.
10. Review agentic features like capital allocation
Every quarter, classify each workflow into one of four states:
- scale: strong economics, reliable outcomes
- optimize: valuable but too expensive or too slow
- constrain: useful only for specific segments or cases
- kill: cost exceeds value and no plausible path fixes it
This is the leadership step most companies skip.
Features with bad unit economics tend to survive because they are technically interesting, strategically fashionable, or attached to executive expectations. A CTO needs a discipline for saying: this workflow should remain assisted, not autonomous; this one should narrow to enterprise accounts only; this one should be removed.
Linear is instructive here culturally. The company is admired not for shipping everything, but for maintaining a high bar on what deserves to exist. AI features need the same product ruthlessness.
05 STRATEGIC TAKEAWAY
Agentic AI should be treated as an operating model decision, not a model selection decision. If you apply bounded execution, task routing, step-level cost attribution, and early escalation, you convert AI from an open-ended cost center into a portfolio of workflows with measurable unit economics. If you do not, the likely outcome this quarter is familiar: rising spend, uneven latency, support exceptions, and leadership pressure to justify why an “AI feature” costs more than the human process it was meant to improve.
06 IMPLEMENTATION ANGLE
Start with one workflow where value is already obvious and human effort is measurable: support triage, account research, document intake, or internal code assistance. Instrument that workflow end to end before broadening autonomy. You need a baseline for median cost, P95 cost, completion rate, escalation rate, and time saved per successful action. Without those numbers, optimization is just taste.
Technically, the current practical stack is straightforward: an orchestration layer with hard limits, a trace pipeline that logs step-level tokens and tool calls, retrieval with aggressive ranking and low top-k defaults, and a policy engine outside the prompt for any action with financial, security, or compliance implications. Tools in the market can support parts of this, but the important move is architectural separation: model reasoning, tool execution, and business rules should not be fused into one opaque loop. related topic
On the team side, assign a single owner for AI unit economics. Not adoption. Not model quality in isolation. Unit economics. In scaling engineering orgs, this often sits between a staff engineer and a product lead who can jointly decide when a workflow should escalate, narrow, or be removed. If your company is growing from 30 to 150 engineers, this is exactly the kind of systems ownership problem that good org design solves early; Amplify can help engineering teams scale, but only if the architecture itself is measurable enough to manage.
