No LLM Proxy, No Observability: Hidden Costs

If every service calls models directly, you lose the control plane that makes AI systems governable.

01 THE PROBLEM

LLM observability debt is the failure mode where model calls are distributed across services, SDKs, and teams without a shared layer for logging, policy, routing, and spend attribution.

The consequence is not abstract. You do not know, with confidence, which product surface caused yesterday’s token spike, which prompt version degraded quality, which customer generated the margin-negative workload, or which fallback path silently shifted traffic from a cheap model to an expensive one.

That gap shows up fast.

Within 30 to 60 days of shipping more than one production AI feature, most teams hit the same wall: usage grows, provider invoices grow faster, and nobody can answer basic operating questions without pulling logs from three systems and manually reconciling request IDs. At that point, “we’ll instrument it later” turns into a tax on every launch, incident, and pricing discussion.

This is why “no proxy” is rarely just an architecture choice. It is usually a governance choice made by accident.

A direct-to-provider setup looks innocent early on. One team calls OpenAI from a Python backend. Another uses Anthropic from a Node service. A third adds Bedrock for data residency or procurement reasons. Product teams move quickly because nothing blocks them. They also create three incompatible telemetry paths, three retry behaviors, three approaches to redaction, and no reliable source of truth.

You can survive that if AI remains an experiment.

You cannot survive it if AI becomes revenue-bearing.

The hidden cost is not only missing dashboards. It is decision latency. When a VP Engineering asks why gross margin dropped after launching an AI copilot, the real issue is not that cost went up. The issue is that nobody can decompose the increase by endpoint, tenant, prompt, feature flag, model, or release train in one place.

That is an observability failure.

And unlike classic infra observability, LLM observability needs to answer a wider set of questions simultaneously:

• What did this request cost?
• What prompt template and version produced it?
• Which model and provider served it?
• What fallback or retry path executed?
• Which user, tenant, or org should the spend be attributed to?
• Was the output good enough?
• Did the request violate policy, latency, or budget thresholds?

Without a proxy or a functionally equivalent control plane, those questions get answered inconsistently, late, or not at all.

Charity Majors has argued for years that observability is about being able to ask new questions of your systems without shipping code first. That framing, associated with Honeycomb’s work, applies even more sharply to LLM systems. If every answer requires adding instrumentation to six repos and waiting for the next deploy, you do not have observability. You have logs.

That distinction matters because LLM systems fail in combinatorial ways.

A prompt change affects token count. Token count affects cost and latency. Latency triggers retries. Retries hit a different region or provider. The fallback model has a larger context window but weaker tool-calling behavior. Quality drops for one customer cohort but not another. Finance sees invoice growth before engineering sees any alert.

A centralized LLM proxy does not solve all of that by itself.

But without one, or without an equally disciplined non-proxy instrumentation strategy, you do not even have the mechanical ability to see it cleanly.

02 WHY IT HAPPENS

The root cause is structural: LLM calls often enter a codebase as application features, but they behave operationally like shared infrastructure.

That mismatch creates the problem.

Product teams think they are integrating an API. In reality, they are adding a new spend surface, a new policy surface, a new reliability dependency, and a new source of sensitive data flow. The org treats the first version as feature work because that is how it gets staffed and prioritized. The failure appears later because the operational burden is cumulative.

This is a classic local-optimization pattern.

Each team chooses the shortest path:

• use the provider SDK directly
• keep prompt templates close to feature logic
• add a few logs
• capture token usage only where the provider exposes it cleanly
• ship

That makes sense for one feature.

It breaks once you have five.

The reason is incentive misalignment. The product team is rewarded for shipping the summarization workflow this sprint. Nobody is rewarded for building a cross-provider cost taxonomy, standardizing prompt version IDs, or making sure retries emit normalized events. Those tasks are platform work, and platform work gets deferred until the pain is undeniable.

Stripe has written extensively about building strong abstractions and internal platforms to reduce repeated complexity across teams. That pattern is relevant here. Once a capability becomes shared, business-critical, and easy to misuse, leaving every team to solve it independently stops being velocity and starts being waste.

The second structural cause is provider fragmentation.

A single SaaS backend might now touch OpenAI, Anthropic, Google, Azure OpenAI, and AWS Bedrock in the same quarter. Each provider returns different metadata, token accounting semantics, request identifiers, model naming conventions, and error categories. Even the definition of “usage” is not consistent enough to support simple cross-provider reporting without normalization.

That means direct integration multiplies semantic drift.

The third cause is that standard APM tools were not designed for LLM-specific dimensions.

Datadog, New Relic, Grafana, OpenTelemetry, and your warehouse are useful, but they do not automatically solve prompt lineage, token cost normalization, provider fallback attribution, or model-level policy enforcement. You still need a canonical event model for LLM requests. Without a control layer, every team invents its own.

The fourth cause is security and compliance pressure.

As soon as legal asks where prompts are stored, whether outputs can contain PII, or whether a regulated customer can be routed only through a specific provider region, the answer cannot be “it depends which microservice made the call.” That answer may work in a prototype. It does not work in a procurement review.

Cloudflare’s public writing on AI Gateway is notable here not because every team should use Cloudflare, but because it captures the real pattern: organizations want a single gateway for reliability, caching, logging, and policy across multiple AI providers. The need exists because provider APIs are not an operational strategy.

There is also a maturity trap.

The first time a team gets burned, they often respond by adding more app-level instrumentation rather than revisiting architecture. That helps temporarily. It does not solve the underlying issue that there is still no shared interception point for routing, budgets, auth, redaction, and standard telemetry.

At that point, the organization has made observability harder in the most expensive way possible: by spreading custom logic into product code.

That debt compounds.

Every new model feature now requires:

• business logic
• prompt work
• evaluation logic
• safety handling
• per-service telemetry glue
• provider-specific error handling
• spend tagging conventions
• local fallback behavior

The direct cost is engineering time.

The larger cost is that your AI platform becomes impossible to reason about as a system.

Will Larson’s writing on staff engineering repeatedly comes back to this principle: systems become brittle when important decisions are distributed without corresponding mechanisms for alignment. LLM integrations are a textbook example. Teams need autonomy to ship. Leadership needs enough centralization to preserve reliability, security, and economics.

A proxy is one of the cleanest ways to create that alignment point.

Not the only way.

But the default alternative is not “same outcomes with less complexity.” The default alternative is fragmented instrumentation and weak controls.

03 WHAT MOST GET WRONG

The most common misdiagnosis is thinking the problem is dashboarding.

It is not.

Teams notice they cannot answer cost or quality questions, so they buy or build more reporting. They pipe provider logs into a warehouse, stitch token counts from invoices, and add Grafana panels by endpoint. Six weeks later they still cannot explain why one release doubled unit cost for enterprise tenants in Europe.

That is because the problem was never the chart layer. It was missing canonical events at the point where requests happened.

A dashboard cannot reconstruct metadata that was never captured.

The second mistake is assuming APM plus structured logs equals observability.

It does not.

You can have immaculate traces and still miss the dimensions that matter for LLM operations:

• prompt template ID
• prompt version
• input and output token counts
• effective model after fallback
• cache hit or miss
• tenant and feature attribution
• redaction status
• evaluation result
• cost in normalized currency units

Without those fields, the traces are operationally incomplete.

The third mistake is treating direct provider SDK usage as “simpler.”

It is simpler for week one.

After that, it is duplication.

Every team reimplements retries, timeouts, fallback rules, spend attribution, and guardrails slightly differently. You end up with ten direct integrations and no confidence that they behave the same way under rate limits or partial outages.

Netflix’s engineering culture has long emphasized paved roads: make the right operational path easier than the custom one. That is the right framing for LLM infrastructure too. If there is no paved road, every engineer creates their own on-ramp to the model layer. The result is not flexibility. It is entropy.

The fourth mistake is swinging too far and forcing all AI traffic through a proxy before the proxy can handle production requirements.

This also fails.

An underbuilt proxy becomes a new bottleneck:

• it adds latency
• it fails open or fails closed unpredictably
• it lacks proper backpressure
• it cannot stream cleanly
• it becomes the least reliable component in the path

Then teams route around it, which is worse than never having one. You now have a partial proxy estate and a false sense of control.

A real example of this pattern exists outside the LLM domain but applies directly: Twitter’s 2023 rate-limit instability, publicly discussed by users and press, showed what happens when foundational request controls are imposed without enough operational resilience. The lesson is not “never centralize.” The lesson is “your control plane must itself be production-grade.”

The fifth mistake is believing no-proxy observability is impossible, so the only choices are “full proxy” or “flying blind.”

That is wrong.

You can build strong observability without a proxy if you enforce a strict instrumentation contract in your app layer. The catch is organizational discipline. Every service must emit the same canonical event schema, with the same IDs, the same prompt metadata, and the same cost normalization. You also need a shared library so teams do not drift.

Ops teams sometimes underestimate how hard this is socially.

OpenTelemetry succeeded because the industry converged on common semantics and libraries. If you want no-proxy LLM observability, you need the equivalent internally. Otherwise each repo becomes its own dialect.

The sixth mistake is focusing only on total spend.

CloudZero has made this point clearly in its AI cost writing: aggregate AI cost is not enough. Finance and engineering both need unit economics. The question that matters is not “what did we spend on LLMs last month?” It is “what did this feature cost per active customer, per workflow, per successful outcome, and did that align with pricing?”

If your observability layer cannot answer that, it is not an executive tool. It is invoice annotation.

The seventh mistake is storing everything.

Teams panic about visibility and dump prompts, completions, and metadata into one giant store. Later they discover they created a privacy and retention problem bigger than the original observability gap. If prompts can include customer data, your logging strategy is now part of your security posture.

OWASP’s guidance on secure design and least exposure applies here directly. Capture what you need for operations. Redact or hash what you do not. Do not make your observability stack the easiest place in the company to exfiltrate sensitive data.

What this costs is measurable, even if the exact dollar amount varies by company.

You pay in four places:

1. Engineering hours

Staff and senior engineers spend cycles answering basic questions with ad hoc SQL and log spelunking instead of improving model quality or product reliability.

1. Gross margin leakage

Expensive model defaults, unnecessary retries, and unobserved prompt inflation can persist for weeks because no one sees the feature-level economics.

1. Incident duration

Mean time to resolution goes up because you cannot isolate whether failures came from one provider, one model family, one deployment, or one customer path.

1. Roadmap drag

Every new AI feature takes longer because teams must reinvent instrumentation, governance, and rollout mechanics.

DORA’s four key metrics are relevant here even though they were not created for AI specifically. If your AI stack lacks a shared control plane, you usually see lower deployment throughput for model-backed features, higher change failure rate, and slower restoration time. The mechanism is straightforward: hidden dependencies produce hidden failures.

04 THE FRAMEWORK

The workable approach is not “proxy everything immediately.”

It is: establish one authoritative LLM control plane, then decide whether that control plane is a network proxy, a shared service abstraction, or a strict application SDK with mandatory event contracts.

For most Series A to C teams shipping multi-feature AI products, a proxy-backed control plane is the cleanest default because it centralizes the hard parts early.

Here is the framework that actually holds up.

1. Decide what “observability” must answer in your business

Start with operating questions, not tools.

If your AI stack is real revenue infrastructure, your observability layer must answer these within minutes, not days:

1. Which features drive LLM spend?
2. Which tenants or customer segments are margin-negative?
3. Which prompt version changed quality or token usage?
4. Which provider or model degraded latency or error rate?
5. Which fallback path was invoked, and how often?
6. Which requests violated policy, budget, or region constraints?

If you cannot answer those six, you do not yet have a production-ready AI control plane.

Write them down and make them your schema requirements.

2. Define a canonical LLM event model before choosing tools

This is the most important technical step.

Every request should produce a normalized event, whether traffic goes through a proxy or not. At minimum, capture:

• request_id
• trace_id
• tenant_id / org_id
• user_id or service actor
• feature_name
• endpoint_name
• prompt_template_id
• prompt_version
• model_requested
• model_served
• provider
• region
• input_tokens
• output_tokens
• cached_tokens if applicable
• latency_ms
• status_code / error_type
• retry_count
• fallback_invoked boolean
• estimated_cost_usd
• policy_flags
• redaction_mode
• eval_score if post-hoc evaluation exists
• release_version

Without prompt version and feature attribution, post-release analysis turns into archaeology.

This is where a proxy helps disproportionately. It can guarantee these fields exist before traffic leaves your boundary.

If you stay no-proxy, build a shared SDK that makes omission impossible. Optional metadata becomes missing metadata under delivery pressure.

3. Separate the data planes: policy, telemetry, and payload retention

Do not treat prompts and completions as a single logging problem.

You have at least three concerns:

• request control: routing, budgets, auth, fallback, rate limiting
• telemetry: normalized event metadata for cost and reliability
• payload retention: whether prompts/outputs are stored, sampled, redacted, or dropped

Conflating these creates either blind spots or compliance risk.

A practical baseline:

• Store full payloads only for sampled traffic or explicit debugging sessions
• Redact obvious PII before persistence
• Keep telemetry events for all requests
• Hash prompt bodies for deduplication where raw retention is not justified
• Define retention windows by environment and customer tier

This pattern mirrors mature observability practice elsewhere. You do not keep packet captures for every request forever. You keep enough structured evidence to operate the system safely.

4. Make cost attribution first-class, not a reporting afterthought

Cost control is observability, not finance plumbing.

For each LLM request, compute or estimate cost at request time. Do not wait for the monthly provider bill. Providers differ in what they expose, so normalization is your job.

Your cost dimensions should support at least:

• cost per feature
• cost per tenant
• cost per active user
• cost per successful workflow
• cost by model and provider
• cost by prompt version
• cost from retries and fallbacks separately

The benchmark that matters is not a universal industry number. It is whether your AI margin profile can be explained. If your head of finance asks why the support copilot cost rose 38% month-over-month, the answer should not require warehouse work.

Pluralsight’s argument on metering is directionally right: central observation is the prerequisite to cost reduction. The exact savings vary, but the mechanism is obvious. You cannot optimize what you cannot attribute.

5. Put routing logic in one place

Routing policy includes more than failover.

It covers:

• default model by feature
• customer-tier overrides
• region constraints
• provider preference
• timeout behavior
• fallback order
• budget-based downgrades
• experiment cohorts

If that logic lives inside product services, every service becomes a mini control plane.

That is exactly what you want to avoid.

Cloudflare’s AI Gateway and similar architectures are useful reference points because they treat provider choice as an operational concern, not feature-local business logic. Even if you do not use a gateway product, copy the principle.

One routing table. One policy engine. One place to audit.

6. Instrument quality, not just infrastructure

This is where many teams stop too early.

Latency and cost matter. They are not sufficient.

You also need request-linked quality signals:

• thumbs up/down
• task completion rate
• regeneration frequency
• human escalation rate
• offline eval score
• schema adherence
• tool-call success rate

For extraction and agentic workflows, schema adherence and tool-call success often matter more than free-form output quality. For writing assistants, user acceptance rate may be the clearest signal.

Vercel’s public work around AI SDK abstractions is instructive here. The value is not only easier integration. The value is creating a common place where model interactions can be instrumented consistently across application surfaces. Consistency is what lets quality and cost be compared honestly.

Tie quality metrics to the same request IDs as spend and latency. Otherwise you cannot make the only decision that matters: whether a more expensive model actually improved outcomes enough to justify the cost.

7. Set operational thresholds before incidents happen

You need concrete budgets and SLO-like thresholds.

Use these as starting points:

• p95 end-user LLM latency target by feature: define it explicitly, often 2–8 seconds depending on interaction type
• error budget for provider/model failures: e.g. less than 1% request failure over 28 days for customer-facing assisted workflows
• fallback invocation threshold: alert if it exceeds a baseline by 2x over 15 minutes
• token growth threshold: alert if average tokens per successful request rises more than 20% week-over-week without a planned prompt change
• margin threshold: flag tenants or features whose LLM cost exceeds expected budget per workflow

For service reliability framing, the Google SRE book is still the best anchor. If a user-facing path matters, define an SLI, set an SLO, and treat excess burn as an engineering event. LLM features do not get a pass because the failure came from a model provider.

8. Build the paved road with escape hatches

Do not ban direct provider usage on day one.

Do make the approved path materially easier.

The paved road should provide:

• unified auth and secrets handling
• standard retries and timeouts
• logging and tracing
• prompt version registration
• automatic cost calculation
• policy checks
• fallback support
• optional payload sampling
• evaluation hooks

Then define an exception process for teams that need direct provider access for a legitimate reason, such as a specialized multimodal endpoint or a provider feature your proxy cannot yet support.

This is how good platform teams win adoption.

Linear is a useful model in a different sense: its product and engineering discipline is famous for reducing optional complexity. The lesson is not about ticketing. It is about system design. Teams adopt the path that preserves speed and predictability. If your AI platform adds friction without obvious value, engineers will bypass it.

9. Decide explicitly between proxy and no-proxy models

Here is the real tradeoff.

Proxy-first model

• Best for centralized policy, telemetry consistency, and routing control
• Faster to achieve organization-wide observability
• Easier to support multi-provider strategy
• Introduces a new component on the critical path
• Requires production-grade reliability and careful latency management

No-proxy, SDK-first model

• Lower network-path complexity
• Can preserve provider-native features and streaming behavior more directly
• Works well if you have strong internal platform discipline
• Harder to enforce complete metadata capture across teams
• More likely to drift across repos and services

A useful threshold: if more than three engineering teams or more than five production features call LLMs, a shared control plane usually pays for itself quickly. Below that, a strict SDK-first approach can work if a single platform owner enforces it tightly.

The failure mode is indecision.

Partial proxying plus inconsistent SDK instrumentation gives you the cost of both models and the benefits of neither.

10. Design the proxy like critical infrastructure, not middleware

If you choose a proxy, treat it as tier-1.

That means:

• stateless where possible
• horizontally scalable
• request timeouts and backpressure defined
• streaming fully supported
• rate-limit aware
• canary releases and rollback path
• explicit degraded modes
• high-cardinality telemetry support
• synthetic checks against each upstream provider
• auth separation between client-facing and provider-facing identities

HashiCorp’s long-running discipline around infrastructure primitives is the right mindset here. The control plane is not a convenience service. It is foundational machinery. Build and operate it accordingly.

11. Keep model governance tied to delivery

Prompt changes should be deployable and auditable like code.

At minimum:

• version prompts
• link versions to releases or config changes
• support staged rollout
• capture before/after token and quality deltas
• require owner metadata for every production prompt
• retain a changelog

GitHub’s engineering and product culture around change visibility is relevant here: if a change affects users, it should be traceable. Prompt edits are no different. Quietly editing a template in production without measurement is the AI equivalent of hot-patching business logic.

12. Review unit economics monthly, not just incidents

Your CTO, finance lead, and product owner should review one page monthly:

• top 10 AI features by spend
• spend per feature versus usage growth
• highest-cost tenants
• quality-versus-cost changes by model
• fallback and retry rates
• major prompt changes and their effect
• candidate workloads for caching or smaller models

This is where observability becomes strategy.

You are no longer asking “what happened?” You are asking “which AI behaviors should we fund, downgrade, reprice, or remove?”

That is the point.

05 STRATEGIC TAKEAWAY

A shared LLM control plane is not infrastructure polish. It is the boundary between an AI product you can operate and one you can only hope is profitable. Apply this early and you get faster incident isolation, cleaner unit economics, and a real basis for model and pricing decisions this quarter. Ignore it and the bill is predictable: within one or two product cycles, your strongest engineers become human middleware between provider logs, finance questions, and product guesses.

06 IMPLEMENTATION ANGLE

The practical path today is straightforward.

If you are early but serious, start with one approved integration layer this month. That can be a lightweight proxy such as an AI gateway pattern, or a mandatory internal SDK if your traffic volume is still low. The non-negotiable part is the schema: every request must emit normalized metadata for feature, tenant, prompt version, model served, tokens, latency, retries, and estimated cost. Pipe those events into your existing observability stack and warehouse before you build fancy dashboards. related topic

If you already have scattered direct integrations, do not try to migrate everything at once. Pick the top two spend-driving features and force them through the new control plane first. Use that migration to harden streaming support, fallback behavior, and redaction rules. Once the platform answers cost-per-feature and cost-per-tenant accurately, adoption gets easier because the value is visible to engineering, product, and finance at the same time.

For teams scaling quickly, this usually lands with a small platform owner group rather than a broad committee. One staff engineer and one product-minded infrastructure engineer can often establish the paved road faster than six teams can align ad hoc. If you are growing the org around that platform, Amplify can help engineering teams scale, but the immediate priority is not headcount. It is architectural clarity and a single source of truth.

07 FAQ

Q: Do you need an LLM proxy to get observability? A: No, but you do need a single control plane. A proxy is the cleanest way to guarantee normalized telemetry, routing policy, and cost attribution across providers. A no-proxy approach can work if every service uses a shared SDK and emits the same canonical event schema; otherwise you get drift, missing metadata, and weak cost visibility. Q: What does an LLM proxy actually add beyond direct provider SDKs? A: An LLM proxy adds centralized routing, standardized logging, policy enforcement, fallback handling, and request-level cost attribution. Cloudflare’s AI Gateway is a public example of this pattern: one layer in front of multiple providers for visibility and control. The technical value is not abstraction for its own sake; it is one place to observe and govern model traffic. Q: What should LLM observability measure in production? A: At minimum, measure request count, latency, error rate, input and output tokens, estimated cost, provider, model served, prompt version, tenant attribution, and fallback rate. The Google SRE book’s SLI/SLO model applies directly: if the feature is user-facing, define reliability and latency targets instead of relying on raw logs. Add quality signals such as task completion, schema adherence, or user feedback so cost can be evaluated against outcomes. Q: Why are direct LLM integrations hard to manage at scale? A: Direct integrations fragment policy and telemetry across codebases. One team handles retries one way, another logs token usage differently, and a third forgets to record prompt version or tenant ID. Stripe’s broader platform lesson applies here: once a capability is shared and business-critical, repeated local implementations create inconsistency and operational drag. Q: When should a startup invest in an LLM proxy or control plane? A: Invest once more than three engineering teams or more than five production features are making model calls, or earlier if AI cost is material to gross margin. That is the point where local integration speed is overtaken by governance, reliability, and pricing complexity. Waiting until finance or enterprise customers ask hard questions usually means you are already carrying observability debt.