The Modern AI Stack Is an Evaluation Stack

Vector DBs, RAG, LangGraph, MCP, and subagents only matter if you can measure, constrain, and improve them.

01 THE PROBLEM

Agentic AI is the failure mode where a system looks capable in demos, then degrades silently in production under real context, real tool access, and real user ambiguity.

That is the central mistake in how teams talk about the modern AI stack.

They discuss vector databases, RAG, LangChain, LangGraph, Model Context Protocol (MCP), skills, subagents, and observability tools like LangSmith as if these are independent product categories to assemble. They are not. They are control surfaces for one problem: getting a probabilistic model to behave reliably enough inside a software system that a user, an engineer, and a business owner can trust it.

The real gap is not “how do I add AI to my app?” It is narrower and more expensive: how do I make a model-grounded system retrieve the right context, choose the right tool, execute the right workflow, and fail safely within an acceptable latency and cost envelope?

If you get that wrong, the timeline to pain is short.

Within two weeks, your team will be debating why the assistant worked in staging and failed for enterprise customers with larger document corpora.

Within a month, support tickets will cluster around three issues: wrong answers with confident tone, nondeterministic behavior across similar prompts, and tool actions that were technically valid but operationally wrong.

Within a quarter, the CTO is staring at a familiar pattern: rising model spend, an unowned prompt layer, no useful eval suite, and no way to separate retrieval errors from reasoning errors from orchestration errors.

This is why the “AI modern tech stack” should be treated less like a frontend-backend-database architecture diagram and more like a reliability architecture.

Vector databases are not the product. RAG is not the product. LangChain is not the product. MCP is not the product.

The product is a bounded system that can answer, decide, and act without surprising you at scale.

That requires a stack built around six hard constraints:

1. Context quality
2. Execution control
3. State management
4. Tool interoperability
5. Evaluation
6. Human override

Miss any one of these, and the rest of the stack turns into expensive middleware around an unmeasured model call.

02 WHY IT HAPPENS

The root cause is architectural mismatch.

Traditional software systems are built around deterministic components. Inputs map to predictable outputs. Failure modes are usually explicit: exceptions, timeouts, bad schema, unavailable dependencies.

LLM systems invert that. The model can produce syntactically correct, semantically plausible, operationally dangerous output even when every surrounding service is healthy.

That changes what “stack” means.

In a conventional SaaS application, the database stores truth, the application layer enforces logic, and the UI presents state.

In an AI system, truth is split across at least four places:

• model weights
• retrieved context
• tool outputs
• workflow state

That split creates a structural problem. No single layer owns correctness.

The vector database might retrieve the right chunk, but the model ignores it.

The model might decide on the right next step, but the tool contract is ambiguous.

The orchestration layer might call the right tools, but the state machine loops or loses user intent.

The MCP server might expose the right capability, but the agent chooses the wrong one because tool selection was under-specified.

This is why agent systems break silently. Every layer is “working,” but the composition is wrong.

The O’Reilly piece on the AI agents stack makes the right point: you need LangGraph, MCP, and evals because agents fail in ways that are not visible through standard application telemetry. That matches what experienced teams discover quickly. CPU, p95 latency, and error-rate dashboards do not tell you whether your agent chose the wrong planning path or retrieved stale policy docs.

A second cause is incentive misalignment inside engineering organizations.

The prototype team is rewarded for visible capability: “it can search our docs,” “it can update Salesforce,” “it can summarize incidents,” “it can write SQL.” The production team inherits the invisible burden: prompt drift, retrieval tuning, cost control, auth boundaries, vendor lock-in, regression testing, and incident response for probabilistic workflows.

This is not new. It mirrors old patterns in distributed systems.

Google’s SRE book formalized the idea that reliability is a feature with explicit budgets. DORA’s four key metrics gave teams a shared way to reason about deployment speed and operational stability. AI systems need the same operational framing, but most orgs have not created it yet.

A third cause is that the ecosystem itself encourages category confusion.

LangChain began as a developer convenience layer. Teams adopted it to move faster, which was reasonable. But convenience abstractions often hide failure boundaries. When the chain fails, where did it fail? Prompt? Retriever? parser? tool schema? retry logic? memory layer?

LangGraph emerged because serious teams needed explicit stateful orchestration rather than hidden control flow. That is the pattern to notice. As systems become more consequential, they move away from magic and toward graphs, state machines, and typed interfaces.

MCP is following a similar path.

The value of MCP is not hype around “AI-native tooling.” The value is standardization. It defines a consistent way for models and applications to access tools and context. That matters because bespoke tool integrations become the next generation of brittle internal APIs if left unmanaged.

The pattern here is familiar to anyone who has lived through service sprawl.

When every team invents its own prompt contract, tool schema, retrieval pipeline, and memory mechanism, you do not get innovation. You get fragmentation.

The final reason this happens: most teams still underinvest in evals because they are unpleasant to build.

They are hard to scope, harder to maintain, and force uncomfortable precision. You have to define what “good” means. You have to write adversarial cases. You have to decide whether faithfulness matters more than verbosity, whether action success matters more than textual elegance, whether partial credit is acceptable in a workflow.

But without evals, every stack decision becomes theology.

Should we use Pinecone, Weaviate, pgvector, or OpenSearch for vector search?

Should we use vanilla RAG, hybrid search, rerankers, or graph retrieval?

Should the workflow live in LangChain or LangGraph?

Should we expose tools via MCP or custom SDKs?

Those are answerable questions only if you can measure retrieval quality, answer quality, tool success rate, and task completion rate on your own workload.

Otherwise you are shopping for architecture by anecdote.

03 WHAT MOST GET WRONG

The common misdiagnosis is simple: teams think they need a smarter model when they actually need a more disciplined system.

That leads to three expensive mistakes.

Mistake 1: Treating RAG as a universal fix

RAG is the failure mode where teams bolt retrieval onto a weak workflow and call it architecture.

RAG helps when the problem is missing or stale knowledge. It does not solve bad task decomposition, weak tool use, missing permissions, or unclear user intent.

A lot of teams discover this the hard way. They index docs, connect a vector database, add a retriever, and watch answer quality improve for fact lookup. Then they try the same stack for multistep execution: “find the latest product requirements, compare them to the shipped feature flags, create a Jira ticket, and draft a customer update.” Retrieval was never the bottleneck there. Workflow control was.

The Udemy-style examples comparing raw LLM output versus RAG output are directionally right. Ask a model about a proprietary term and it hallucinates; add retrieval and it grounds the answer. But those examples often stop at single-turn QA. Production systems fail on compound tasks, permissioned data, and state across sessions.

Mistake 2: Using LangChain where a state machine is required

LangChain is useful for composition. It helps glue models, prompts, retrievers, and tools together. It is good for getting an application running quickly.

It is not the right abstraction for every production agent.

Once your system needs branching logic, retries based on semantic outcomes, human approval gates, resumability, or subagents with bounded responsibility, hidden chain flow becomes a liability.

This is why LangGraph matters. It makes state explicit. You can model a workflow as nodes and edges instead of hoping a sequence of chained calls behaves under stress.

This is the same maturity move that backend teams make when cron jobs and ad hoc scripts become Airflow DAGs, Temporal workflows, or explicit state machines.

Shopify’s engineering teams have written extensively about making system behavior explicit in high-scale infrastructure and product systems. The principle applies here too: hidden control flow is tolerable in prototypes and expensive in production.

Mistake 3: Treating tool access as a UX feature instead of a security and reliability boundary

MCP is often pitched as “USB-C for AI,” which is memorable but incomplete.

The important point is not convenience. It is governance.

If your agent can access GitHub, Slack, Postgres, Stripe, HubSpot, and internal admin actions, then tool invocation becomes a production interface with all the same concerns as any internal platform:

• authentication
• authorization
• versioning
• rate limiting
• observability
• schema evolution
• blast radius

Ignore this and the system becomes impossible to reason about.

Cloudflare’s engineering and product work on AI tooling has consistently emphasized developer control at the platform boundary. That is the right instinct. Standardized access only helps if the platform team can see and constrain it.

Mistake 4: Building a “general agent” before defining skills

A skill is a bounded capability with a clear contract, known inputs, known tools, and measurable outputs.

Most teams skip this layer. They jump from “chat with tools” to “autonomous agent.”

That is backwards.

The systems that work in production usually decompose into skills first:

• retrieve policy documents
• summarize account history
• compare code diffs
• classify inbound tickets
• draft an incident report
• execute a refund under policy constraints

Only after those skills are stable do teams compose them into broader agents or subagents.

This mirrors how Stripe and GitHub approach internal platform capabilities more generally: narrow interfaces, explicit ownership, and repeatable abstractions beat broad magical surfaces.

Mistake 5: No evals, only anecdotes

This is the most common and the most damaging.

A founder says, “I tried it five times and it seems good.”

A PM says, “The demo worked in the sales call.”

An engineer says, “The trace looks okay.”

None of those are evals.

LangSmith exists because tracing, testing, and evaluation need to be first-class. You need datasets, expected outcomes, human feedback loops, regression comparisons, and per-step visibility.

The alternative is familiar to anyone who has run on-call for a brittle system: users become the test suite.

And we have enough public examples of what happens when software ships without a sufficient evaluation discipline. Microsoft’s early public rollout issues with Bing Chat demonstrated how quickly model behavior can diverge in real user environments, especially over long sessions and unbounded interaction patterns. The lesson was not “never ship AI.” The lesson was “constrain the system you actually have, not the one your demo suggests.”

04 THE FRAMEWORK

The stack that actually works is not “pick the best tools.” It is “sequence the architecture in the order risk appears.”

Here is the framework I recommend for CTOs and Staff+ engineers building a modern AI application stack today.

1. Start with tasks, not models

Define the top 5 to 10 user tasks the system must complete.

Not “customer support agent.” Not “AI copilot.” Not “research assistant.”

Write the tasks in operational language:

1. Answer policy questions from approved documentation with citations.
2. Summarize the last 30 days of account activity and list anomalies.
3. Draft but do not send customer replies for billing disputes.
4. Create Jira tickets from incident postmortems with prefilled severity and owner fields.
5. Query internal analytics via an approved SQL generation path and return both SQL and answer.

This sounds basic, but it avoids the biggest source of architectural waste: building generalized orchestration before you know what needs orchestrating.

For each task, define:

• success output
• acceptable failure mode
• allowed tools
• expected latency
• whether human review is mandatory

If you cannot do this, you are not choosing a stack. You are buying optionality you have not earned.

2. Separate knowledge retrieval from action execution

RAG and tool use should not be treated as the same thing.

RAG answers: “What context should the model see?” Tools answer: “What can the model do?”

Keep those layers distinct in both architecture and evaluation.

A practical split:

• Retrieval layer: embeddings, chunking, indexing, metadata filters, hybrid search, reranking
• Execution layer: tool schemas, auth scopes, side effects, confirmation gates, retries

Why this matters: if a system answers incorrectly, you need to know whether retrieval missed the needed document, the model misread the document, or the workflow invoked the wrong tool.

This is where vector databases fit.

Use a vector database when semantic retrieval quality is meaningfully better than keyword search for your corpus and query patterns. That is often true for internal docs, tickets, wikis, transcripts, and product specs. It is less universally true for highly structured operational data, where SQL or filtered search is often the right path.

Do not store everything in a vector DB by default.

Use these rough guidelines:

• If the source of truth is relational and updated frequently, start with direct query tools.
• If the corpus is large, text-heavy, and semantically diverse, add vector search.
• If exact matches and semantic matches both matter, use hybrid retrieval.
• If ranking quality drives user trust, add reranking before changing the embedding model.

This is one reason teams choose Postgres + pgvector early. It reduces system count and keeps metadata joins simple. As scale, latency, or retrieval complexity increases, they may move to specialized systems like Pinecone, Weaviate, Qdrant, or OpenSearch-based hybrid setups.

There is no prize for adopting a specialized vector database too early.

3. Use RAG for grounding, not memory

This distinction saves months.

RAG is retrieval over a knowledge base. Memory is persisted interaction or state across sessions.

Do not confuse the two.

A user’s preferences, prior approvals, unresolved workflows, and pending review states belong in application state, not “just retrieve from the conversation history.”

Long-term memory sounds elegant until you need auditability, deletion guarantees, or deterministic replay.

Use explicit state for:

• user profile
• workspace configuration
• prior approvals
• execution checkpoints
• workflow status
• compliance-relevant events

Use RAG for:

• docs
• knowledge articles
• transcripts
• past incidents
• product specs
• codebase references where retrieval helps

This is exactly why stateful orchestration frameworks matter. LangGraph is useful because it lets you model memory and workflow state deliberately instead of hoping prompt stuffing behaves like application logic.

4. Graduate from chains to graphs as soon as workflows branch

LangChain is a productivity layer. LangGraph is an execution discipline.

Use LangChain for:

• prototyping
• simple prompt/retrieval/tool compositions
• lightweight developer ergonomics

Use LangGraph when you need:

• explicit workflow state
• branching decisions
• retries by node
• pause/resume
• human-in-the-loop checkpoints
• subagents with scoped roles
• auditability of execution path

This is not theory. It is the standard maturity path for systems with side effects.

A useful threshold: if a workflow can call more than 3 tools, can loop, or can trigger a user-visible side effect, move to an explicit graph.

That threshold is intentionally low. Hidden control flow gets expensive faster than most teams expect.

5. Define skills before agents

This is the most underused pattern in the AI stack.

A skill is a tested unit of capability. It should be callable, observable, and replaceable.

A good skill has:

• a narrow objective
• fixed input schema
• bounded tool access
• explicit output schema
• pass/fail criteria
• owner

Examples:

• `summarize_prd`
• `compare_contract_versions`
• `lookup_account_risk_flags`
• `draft_refund_response`
• `extract_action_items_from_meeting`

These can be implemented as prompt + retrieval + tool logic, or as graph nodes, or as callable services.

Once skills are stable, you can compose them into subagents.

A subagent is just a workflow-scoped actor with limited responsibility and constrained tools. Not a tiny AGI. Not a mystical planner. A bounded execution unit.

Useful subagents often include:

• retrieval subagent
• code analysis subagent
• support-policy subagent
• data-query subagent
• action-approval subagent

The reason this works is simple: decomposition makes evals possible.

You can test a `policy_lookup` skill independently of the broader customer support workflow. You can test a `sql_generation` skill independently of the analytics narrative layer.

That is how software gets better over time.

6. Use MCP to standardize tool access, not to avoid architecture

MCP is worth adopting when your organization has more than a handful of tools or expects multiple model clients, agent runtimes, or product surfaces to share the same capabilities.

The right use case is not “we want the latest protocol.”

The right use case is:

• we need a standard interface for tools and context
• we want to expose capabilities to multiple applications
• we need centralized auth and audit patterns
• we want to reduce one-off adapter code

Treat MCP servers like internal platform APIs.

That means:

• typed schemas
• clear permission boundaries
• versioning policy
• observability
• fallback behavior
• owner per server

If an MCP server can trigger side effects, require confirmation or policy checks upstream.

Do not let “the model decided to call the tool” substitute for authorization logic.

7. Make evals the backbone of the stack

This is the section most teams should read twice.

Before selecting vendors, define your evaluation harness.

“Harness” here means the system that runs your test cases, compares outputs, scores behavior, and reports regressions across prompts, models, retrievers, tools, and workflows.

A useful eval harness includes:

• representative datasets
• golden examples
• adversarial examples
• per-step traces
• answer-quality scoring
• tool success/failure scoring
• cost and latency tracking
• regression comparison by release

This is where LangSmith earns its place. It gives teams tracing, experiment comparison, and evaluation workflows around LangChain/LangGraph-based systems. Even if you do not standardize on LangSmith, you need the capability category it represents.

Measure at three levels:

Retrieval metrics

• recall@k on known-answer sets
• citation accuracy
• chunk relevance
• reranker impact

Generation metrics

• faithfulness to sources
• schema validity
• refusal behavior
• completeness against task rubric

Workflow metrics

• task completion rate
• tool-call success rate
• human-escalation rate
• p95 end-to-end latency
• cost per successful task

For operational framing, borrow from SRE and DORA thinking.

Google SRE popularized error budgets because reliability needs an explicit threshold for imperfection. Do the same here.

Examples:

• unsupported or hallucinated answer rate under 2% on internal support tasks before wider rollout
• tool execution success above 95% for read-only actions before enabling write actions
• p95 latency under 8 seconds for chat-based retrieval tasks
• human-review rate under 30% before claiming the workflow is materially automating work

Those numbers are not universal truths. They are examples of how to turn capability into an operating contract.

If you do not set thresholds, every release becomes an argument.

8. Keep side effects behind approval boundaries

Write actions are qualitatively different from read actions.

Reading a document and summarizing it is one risk class.

Sending an email, modifying a record, issuing a refund, deleting a file, or merging code is another.

You should treat action autonomy as a maturity ladder:

Stage 1: recommend only Stage 2: draft + human approve Stage 3: auto-execute low-risk actions Stage 4: auto-execute bounded workflows with rollback paths

Most teams should spend longer in Stage 2 than they expect.

The highest-value pattern in the first 6 months is not “full autonomy.” It is “high-quality draft generation inside a human-owned workflow.”

That is where trust forms.

GitHub Copilot’s adoption arc is instructive. It succeeded first as assistive generation inside a developer-controlled loop, not as autonomous software delivery. The market lesson is straightforward: augmentation beats autonomy until the evaluation and rollback story is strong.

9. Design for observability by step, not just by request

Traditional application traces often treat a request as one transaction.

Agent systems need finer granularity.

You need to inspect:

• prompt inputs
• retrieved docs
• model outputs
• tool selection rationale
• tool inputs/outputs
• graph transitions
• retries
• final user-visible response

This is why observability tools in this stack are not optional.

A single bad answer can come from:

• stale index
• bad chunking
• missing metadata filter
• prompt regression
• model version change
• malformed tool schema
• tool timeout
• graph branch misfire

Without step-level traces, debugging turns into folklore.

Datadog, Cloudflare, and Vercel have all pushed the broader industry toward tighter developer feedback loops in modern application systems. AI stacks need the same discipline, but with semantic traces layered over standard infra traces.

10. Optimize for operability, not framework purity

The right stack is the one your team can run.

A pragmatic default for a Series A–C company might look like this:

• foundation model API from Anthropic, OpenAI, or Google
• application database in Postgres
• pgvector first, specialized vector DB later if needed
• object storage for source docs
• LangChain for composition
• LangGraph for stateful workflows
• LangSmith or equivalent for tracing/evals
• MCP servers for key internal tools
• explicit skill library with owners
• approval UI for all write-side actions
• evaluation harness in CI for every workflow release

This stack is not elegant in the abstract. It is operable.

That matters more.

Linear’s product and engineering reputation comes from disciplined simplicity and performance-oriented decisions. The lesson is not “copy Linear’s exact stack.” The lesson is that operational clarity compounds. AI systems need the same bias.

05 STRATEGIC TAKEAWAY

The decisive architecture choice is this: build your AI system as a controlled workflow platform, not as a chat interface with extra APIs. If you do that, vector databases, RAG, LangGraph, MCP, skills, subagents, and LangSmith each have a clear role and owner. If you do not, the next quarter will be spent arguing over model quality while the real issues are retrieval drift, weak state handling, and zero eval discipline. For a CTO making platform decisions this quarter, that difference shows up fast in three numbers that actually matter: cost per successful task, human-review rate, and time-to-debug regressions after each release.

06 IMPLEMENTATION ANGLE

If I were standing up this stack in a 20–200 person company today, I would not start by hiring “agent engineers” or buying five AI infra vendors. I would start with one product surface, three narrow workflows, and an eval harness built before broad rollout. The first team would usually be one Staff+ engineer, one senior product engineer, one infra/platform engineer part-time, and one PM who can define task rubrics precisely. That team owns the skill contracts, traces, approval boundaries, and release criteria.

The technical rollout should happen in layers. First, build read-only workflows with retrieval and citations. Second, instrument every step with traces and create a regression dataset from real usage. Third, introduce tool use through tightly scoped MCP servers or direct tool adapters. Fourth, move any branching workflows into LangGraph. Fifth, permit write-side actions only behind human approval until the task completion rate and tool success rate are stable for at least a few release cycles. related topic

If the organization is growing quickly, this becomes a platform problem earlier than expected. The pattern that emerges at scale is that application teams need reusable skills, shared eval tooling, and common policies for auth and side effects. That is where a lightweight AI platform team earns its keep. If you are also figuring out how to scale the engineering organization around this, Amplify can help teams think through platform ownership, hiring shape, and execution design without forcing a vendor-first architecture discussion.

07 FAQ

Q: What is the modern AI tech stack for production applications? A: A production AI stack usually includes a foundation model, a retrieval layer such as RAG, a vector database or hybrid search system, an orchestration layer like LangChain or LangGraph, a tool-access layer such as MCP, and an evaluation/observability layer such as LangSmith. The critical difference from a demo stack is that production systems also need explicit state, approval boundaries, and regression testing, which is why O’Reilly’s “AI Agents Stack” emphasizes evals and workflow control. Q: When should a team use a vector database instead of PostgreSQL? A: Start with PostgreSQL plus pgvector when your corpus is modest, metadata joins matter, and your team wants operational simplicity. Move to a specialized vector database like Pinecone, Weaviate, or Qdrant when retrieval scale, indexing behavior, filtering complexity, or latency requirements exceed what your Postgres setup can comfortably handle. The right decision depends less on hype and more on retrieval quality and operability on your own workload. Q: What is the difference between LangChain and LangGraph? A: LangChain is primarily a developer framework for composing model calls, retrievers, prompts, and tools. LangGraph is for stateful, graph-based orchestration where workflows need branching, retries, pause/resume, human approvals, or subagents. A useful practical rule is to stay with LangChain for simple compositions and move to LangGraph once a workflow branches or triggers side effects. Q: What problem does MCP actually solve? A: MCP, or Model Context Protocol, standardizes how AI applications access tools and context sources. Its real value is not novelty; it is governance and reuse. By exposing capabilities through a consistent protocol, teams can centralize auth, versioning, and observability instead of maintaining fragile one-off integrations for each model client or agent runtime. Q: How should teams evaluate RAG and agent systems before production? A: Teams should evaluate at three layers: retrieval quality, model output quality, and workflow success. That means measuring things like recall@k for retrieval, faithfulness and citation accuracy for answers, and task completion rate, tool-call success, latency, and human-review rate for end-to-end workflows. This approach mirrors the reliability mindset in Google’s SRE book and the measurement discipline behind DORA metrics: if it matters in production, it needs an explicit threshold before rollout.