AI coding gains are real locally, but without system redesign they turn into throughput illusions.
01 THE PROBLEM
The AI benchmark illusion is the failure mode where coding tools score well on narrow tasks, developers produce more code-shaped output, and engineering productivity still does not improve at the system level.
That gap is now large enough to matter financially.
Teams buy copilots, roll them out in a quarter, and see immediate local signals: more pull requests, more lines changed, faster first drafts, shorter time-to-first-commit. Six months later, the roadmap is still late, review queues are longer, flaky tests are up, and senior engineers are spending more time validating machine-generated work than moving architecture forward.
This is not a theoretical objection to AI coding tools. It is a measurement problem.
The problem starts when leadership confuses benchmark performance with production performance, and individual speed with organizational throughput. A model can solve a LeetCode-style function, autocomplete boilerplate, or generate a migration script. None of that proves the team ships customer value faster, with fewer incidents, or with lower cognitive load.
The strongest public signal is not “AI doesn’t help.” It is “AI helps in places that most dashboards overvalue.”
Faros AI reported in 2025 that engineers using AI tools merged 98% more pull requests, while review time increased 91%. That is the shape of the issue in one sentence: more artifacts, slower system flow. LinearB, drawing on data from 8 million pull requests, made a similar point: high adoption rates create an illusion of progress, but they do not prove value creation.
If you lead engineering, this matters because your bottleneck has shifted.
For most software teams, coding was never the only bottleneck. It was one stage in a chain that includes specification clarity, dependency management, review quality, test reliability, deployment safety, incident response, and customer feedback loops. AI compresses one part of that chain. Unless you redesign the rest, work simply piles up downstream.
This is why engineering productivity can “stall” even while developers feel faster.
The subjective perception is understandable. Developers spend less time on syntax recall, scaffolding, and repetitive glue code. GitHub’s research around Copilot has consistently shown developers reporting higher satisfaction and faster task completion on certain workflows. But self-reported speed is not the same as delivery performance. The DORA framework never measured output by volume of code; it measured deployment frequency, lead time for changes, change failure rate, and time to restore service because these are system outcomes, not local activity metrics.
That distinction is where most AI productivity conversations break down.
A benchmark tells you whether a model can complete a bounded task under controlled conditions. An engineering organization needs to know whether that capability improves the throughput and reliability of a socio-technical system with real constraints: legacy services, undocumented dependencies, compliance requirements, uneven test coverage, on-call burden, and humans with limited review bandwidth.
Those are different questions.
The real-world consequence is predictable. In the first 30 to 90 days, AI tools produce visible velocity. In the next two quarters, review and QA become the choke points. By month nine, staff engineers are quietly absorbing the tax: architecture drift, repeated regressions, duplicated patterns, weaker abstractions, and more time spent distinguishing “works in a demo” from “safe to maintain.”
That is why the benchmark illusion is dangerous. It gives leaders permission to declare productivity gains before the operating system of the engineering org has actually improved.
02 WHY IT HAPPENS
This happens because AI compresses code generation faster than organizations can compress decision-making, validation, and operational risk.
That is the structural reason.
Most benchmarks isolate the model from the system around it. They score correctness on narrowly scoped tasks, often with known interfaces, static requirements, and no organizational context. Production software is the opposite. Requirements move, interfaces are ambiguous, and the hard part is often not writing code but deciding what code should exist, where it should live, who should approve it, how it gets tested, and what tradeoffs it imposes six months later.
A coding assistant can accelerate token production. It cannot resolve unclear ownership boundaries, fix weak service contracts, or make a brittle CI pipeline trustworthy.
That matters because throughput in engineering is set by the slowest constrained stage, not by the fastest optimized stage. This is basic queueing behavior, and every mature engineering org eventually rediscovers it. Increase arrivals into a constrained review or test system and cycle time grows. You do not get linear gains. You get congestion.
That is exactly what the Faros AI finding suggests: a near doubling of merged PRs paired with a 91% increase in review time. More work entered the system than the validation layer could absorb.
The second reason is incentive misalignment.
Developers are rewarded, formally or informally, for visible progress. AI tools are excellent at creating visible progress. You get fuller diffs, quicker first drafts, and more active branches. Leaders under pressure to justify spend can point to adoption rates, suggestion acceptance, or increased pull request volume. Vendors reinforce this because these metrics are easy to capture and easy to market.
But these are upstream signals.
DORA’s four key metrics became influential precisely because they connect engineering behavior to delivery outcomes. Nicole Forsgren, Jez Humble, and Gene Kim argued in Accelerate that high performance comes from optimizing the system for fast, safe flow. Not code volume. Not hours typed. Not “AI interactions per engineer.”
The third reason is architectural asymmetry.
AI tools work best where patterns are repetitive, context windows are sufficient, and correctness can be cheaply verified. They work worst where systems are highly coupled, interfaces are underdocumented, and a small mistake carries large blast radius. That means AI often adds the most code in the parts of the codebase that were already easiest to change, while your actual delivery risk sits in stateful data systems, infra glue, auth boundaries, migrations, and edge-case business logic.
A staff engineer recognizes this immediately: generation scales faster than judgment.
That asymmetry creates a hidden burden on senior people. Junior and mid-level engineers can produce more output with AI. Someone still has to review whether it matches service conventions, security constraints, failure semantics, retry logic, idempotency expectations, observability standards, and backwards-compatibility rules. If these rules are not encoded in tooling, they become review labor.
Review labor does not scale cheaply.
Stripe has written for years about building guardrails into APIs, testing, and operational workflows because reliability comes from systemized constraints, not heroics. The same logic applies here. If AI increases the rate of code creation without a corresponding increase in automated policy checks, integration tests, ownership clarity, and deploy safeguards, you shift burden onto the most expensive humans in the loop.
The fourth reason is that most organizations still treat software development as “write code, then inspect it,” while AI requires “constrain intent, then verify behavior.”
That sounds abstract, but the operational difference is concrete. In a pre-AI workflow, a reviewer can often infer thinking from the shape of code because humans type slowly enough that output is naturally filtered. With AI assistance, output arrives pre-expanded. You see ten times more implementation surface area from the same level of underlying understanding. Review shifts from reading code to auditing whether the developer understood what they accepted.
That is a harder job.
GitHub’s own framing around Copilot has generally emphasized task assistance and developer experience, not a blanket claim that all organizations will ship proportionally faster. That distinction matters. Better local ergonomics are real. System productivity gains require organizational adaptation.
The fifth reason is benchmark transfer failure.
A benchmark may show that a model can generate a correct API endpoint, unit test, or SQL query. In production, the endpoint needs feature flags, audit logging, rate limits, auth scopes, tracing, runbooks, dashboards, fallback behavior, migration sequencing, and rollback safety. The “benchmark-complete” portion may only represent 20% of the real work.
This is why leaders who only ask, “Can the model write code?” get misled.
The better question is, “Which stages of our engineering system become faster, and which become more expensive, when code generation gets cheaper?”
That answer is usually uncomfortable.
Requirements quality becomes more important because AI amplifies ambiguity. Review quality becomes more important because low-cost code inflates verification demand. Test signal becomes more important because brittle or noisy CI cannot absorb higher code volume. Architectural consistency becomes more important because models tend to imitate local patterns, good and bad. Production telemetry becomes more important because regressions can arrive faster.
Netflix’s engineering culture has long emphasized paved roads, strong delivery tooling, and production visibility. The reason is not aesthetic. It is throughput economics. Standardized paths reduce decision overhead and make safe change easier at scale. AI coding tools increase the value of that approach because generated code performs better when the environment is opinionated.
The pattern that emerges at scale is simple: AI boosts output most in organizations that already invested in constraints.
Everyone else gets a surge in code and a slower path to confidence.
03 WHAT MOST GET WRONG
Most teams misdiagnose this as a model quality problem.
They say the assistant hallucinates, the code is noisy, or the generated tests are weak. All of that can be true. It is also not the main issue.
The main issue is operational design. Teams try to fix a system bottleneck with a tool upgrade. They switch vendors, tune prompts, or buy enterprise seats from a different provider while the actual bottlenecks remain unchanged: poor code ownership, long-lived branches, inconsistent review standards, slow CI, and no clear definition of “done” beyond passing unit tests.
That fails because better generation only increases arrival rate into the same constrained system.
A second common mistake is measuring the wrong layer.
Leaders track prompt counts, suggestion acceptance, commits per engineer, lines changed, or pull requests opened. These metrics rise quickly, which creates false confidence. But they are not meaningful unless paired with downstream measures: median review time, PR reopen rate, escaped defect rate, change failure rate, deployment frequency, and time-to-restore.
LinearB’s public argument on 8 million PRs lands here: adoption is not value. If AI increases coding throughput but worsens review latency or deployment risk, you have not improved productivity. You have increased local activity and moved cost downstream.
A third mistake is assuming the best engineers benefit the most from AI in all contexts.
They often do not.
Senior engineers spend disproportionate time on problem framing, architecture, sequencing, migration strategy, and edge-case reasoning. Those are the least benchmark-friendly tasks. In some environments, AI increases their burden because they inherit more review work, more consistency enforcement, and more cleanup of generated implementations that are superficially plausible but misaligned with system intent.
This is one reason the “everyone is 30% faster” narrative breaks in practice. The gain is uneven across roles and workflows.
A fourth mistake is believing that more granular output means more parallelism.
It often means more fragmentation.
Teams encouraged to keep PRs small may start producing a larger number of AI-assisted changes. If ownership boundaries are weak, this creates review thrash. Related changes are split across multiple services, reviewers lose system context, and integration risk increases. What looked like faster iteration turns into more coordination overhead.
Airbnb and Shopify have both written extensively about platform consistency and internal developer systems because scaling product delivery requires reducing variation, not just increasing motion. AI tends to amplify whatever variation already exists. In a disciplined codebase, that can be useful. In a fragmented codebase, it multiplies inconsistency.
The fifth mistake is treating code review as the permanent quality filter.
That was already a weak assumption before AI.
Code review is good at catching obvious logic errors, readability issues, and design mismatches. It is bad at proving runtime behavior across distributed systems, hidden state transitions, security edge cases, and concurrency failure modes. If your response to AI-generated code is “we’ll just review more carefully,” you have chosen the most expensive and least scalable control point available.
Google’s Site Reliability Engineering guidance repeatedly emphasizes reducing toil through automation and designing for measurable reliability. Human review alone is not a sufficient control mechanism for high-rate change systems. AI increases the rate of candidate changes. Therefore, human review must become narrower, more targeted, and better supported by automation.
The sixth mistake is assuming this is a temporary adoption dip.
It is not, unless you redesign the surrounding system.
Organizations often tell themselves that once engineers get better at prompting, gains will materialize. Prompt skill helps at the margin. It does not solve weak test suites, low observability, or unclear API contracts. A mediocre engineer with strong guardrails will usually outperform a prompt expert operating inside a brittle, low-signal delivery system.
The cautionary example here is not one single AI postmortem. It is the long history of local optimization in software engineering.
DORA and Accelerate became important because many organizations had already learned this the hard way. Teams optimized for utilization, approval rigor, and large planned releases, only to make delivery slower and riskier. AI coding tools can replay the same mistake in a modern form: optimize for generated output and feel productive while the system gets harder to operate.
The specific cost shows up in three places.
First, senior time. Staff+ engineers become throughput governors rather than force multipliers.
Second, quality variance. More code enters production with inconsistent assumptions, especially around retries, auth, data handling, and observability.
Third, roadmap integrity. Teams commit to AI-adjusted delivery expectations before understanding where the actual bottlenecks moved.
That is how benchmark optimism turns into planning debt.
04 THE FRAMEWORK
The approach that works is to treat AI coding as a throughput amplifier that must be constrained by delivery-system design.
Not “roll out copilots and hope.” Not “ban AI until models improve.” Build an operating model where cheap code does not create expensive uncertainty.
Here is the framework.
1. Measure system outcomes before local activity
Start by establishing a baseline on delivery metrics that actually matter.
Use the DORA four key metrics as the core:
- Deployment frequency
- Lead time for changes
- Change failure rate
- Time to restore service
Then add AI-era companion metrics:
- Median PR review time
- PR reopen rate after review
- Escaped defect rate within 7 or 14 days of merge
- CI pass reliability, especially flaky test rate
- Percent of changes with full ownership coverage
- Mean time from first draft to production, not first draft to PR
If your team cannot produce these metrics weekly, you do not have enough visibility to claim productivity gains.
The threshold to watch first is review delay. If pull request volume rises and median review time rises more than 20% over baseline for two consecutive sprints, AI is already overfeeding a constrained validation system. That threshold is a practitioner heuristic, not a universal law, but it is operationally useful because review queues compound quickly.
DORA’s work matters here because it gives you the right frame: optimize for flow and stability together. If deployment frequency goes up but change failure rate also rises, you have not created productivity. You have converted engineering time into incident risk.
The Real Cost of Hiding Salary Ranges in Engineering Job Posts2. Segment work by verification cost
Do not ask, “Where can AI write code?”
Ask, “Where can we verify correctness cheaply?”
This one change fixes a surprising amount.
Classify work into three buckets:
Low verification cost
- UI scaffolding
- test fixture generation
- documentation
- typed adapters
- internal tooling
- repetitive CRUD paths with strong contracts
Medium verification cost
- service endpoints with stable dependencies
- well-covered backend business logic
- migration helpers under supervision
- analytics pipelines with backfill checks
High verification cost
- auth flows
- payments
- distributed state transitions
- concurrency-sensitive code
- infra provisioning
- schema changes with irreversible blast radius
- security-critical paths
AI should be used aggressively in the first bucket, selectively in the second, and with strict controls in the third.
This is where mature teams get leverage while avoiding theater. Cheap verification, not model capability, determines whether acceleration is real.
Cloudflare’s engineering writing repeatedly shows the value of strong automated testing, staged deployments, and production safeguards in a globally distributed environment. The lesson is not “Cloudflare uses AI this way.” The lesson is that high-change systems remain safe only when verification pathways are explicit and layered. AI should follow those pathways, not bypass them.
3. Shrink review scope, not just PR size
Many teams respond to AI-generated output by saying “keep PRs small.” That is directionally right and often operationally wrong.
Small diffs can still have large review scope if they cross ownership boundaries, hide generated complexity, or package multiple semantic changes together.
Redefine reviewability with stricter rules:
- One behavioral change per PR
- One owning team per PR where possible
- Description must state intent, risk, and rollback path
- Generated code must be identified when it exceeds a threshold, for example 30% of changed lines
- Reviewers approve behavior and interface decisions, not every token
This only works if your tooling supports it.
GitHub’s code review model is flexible enough to build around, but the process discipline has to come from the organization. If generated code arrives in large slabs with weak explanations, reviewers cannot assess intent efficiently. You get superficial approvals or endless back-and-forth. Neither scales.
The practical trick is to review at the abstraction boundary:
- contract change
- failure behavior
- observability
- security posture
- migration safety
If a team is still line-editing 800 lines of generated glue code, it has already lost.
4. Move quality controls left into automation
Human review should validate intent and exceptions. Tooling should validate conformance.
That means investing in:
- stronger static analysis
- policy-as-code for security and compliance
- API contract tests
- snapshot and golden tests where appropriate
- required observability hooks
- CI rules for ownership and dependency boundaries
- linting for framework and architecture conventions
This is where companies with strong platform engineering discipline pull away.
Shopify has written about standardized developer environments and internal tooling that reduce variation at scale. Figma and Stripe have both emphasized the importance of reliability and consistency in their engineering systems. The common pattern is clear: define the path, automate the constraints, and make the safe choice the default.
AI raises the ROI of this work because every automated guardrail scales across all generated output.
One concrete metric: if more than 5% of test failures in CI are due to flaky tests, AI-driven throughput gains will degrade rapidly. That 5% is a practical red line used in many mature teams because above it, confidence in CI falls enough that humans start rechecking what the pipeline should have resolved. Once confidence drops, review latency and merge hesitation rise.
5. Encode architecture so the model imitates good patterns
AI does not reason about your architecture the way your principal engineer does. It imitates local examples and whatever context you provide.
So give it better context.
Create machine-usable engineering standards:
- service templates with approved defaults
- canonical examples for auth, retries, logging, metrics, tracing
- ADR summaries linked in repos
- codegen-safe module boundaries
- “do not use” and deprecation markers enforced in tooling
- reference implementations for common flows
Vercel’s engineering culture and product design have consistently leaned into opinionated defaults because they reduce cognitive overhead and produce more consistent outcomes. That same logic applies internally. AI performs better in environments with a paved road.
Without that, it learns your accidents.
This is one of the least appreciated tradeoffs in AI adoption. Highly flexible architectures feel empowering to senior engineers but produce weak training context for AI-assisted development. Opinionated systems can feel restrictive, but they generate more reliable assisted output.
If you want AI speed, you usually need more architectural standardization than your team is currently comfortable with.
6. Rebalance the role of senior engineers
If AI increases junior and mid-level output, senior engineers cannot remain primarily ad hoc reviewers.
That role does not scale.
Shift staff+ time into:
- defining paved roads
- codifying review heuristics
- identifying high-risk change classes
- improving test signal
- setting service-level expectations for delivery systems
- pruning bad abstractions before they become training examples
Will Larson’s writing on engineering leadership repeatedly points to leverage coming from systems, not individual intervention. AI makes this harsher. Every undocumented exception now gets cloned faster. Every clear standard gets reused faster.
A practical rule: if staff engineers spend more than 25% of their week on routine PR review for AI-assisted work, your controls are in the wrong place. Move the burden into templates, checks, and standards.
7. Run controlled experiments by workflow, not by seat count
Do not ask whether “AI works for engineering.”
Test whether it improves specific workflows under measurable conditions.
Examples:
- API endpoint implementation in a service with >80% integration test coverage
- refactoring typed frontend components in a design-system-aligned repo
- writing migration scripts with mandatory dry-run verification
- creating internal support tooling with low production blast radius
For each workflow, define:
- baseline cycle time
- baseline defect rate
- review time
- rollback or hotfix frequency
- engineer satisfaction
Run experiments for 4 to 6 weeks. Keep the unit of analysis stable. Compare against a matched control workflow if possible.
This is boring compared with an all-hands rollout. It is also how you avoid spending six figures on licenses while teaching your organization the wrong lessons.
PostHog and Supabase are good examples of engineering cultures that move quickly in public while remaining practical about iteration loops, product feedback, and developer experience. The lesson is not to mimic their stack. It is to validate workflow gains in context rather than extrapolating from generic benchmarks.
8. Plan capacity around verification, not generation
Once AI adoption rises, your staffing assumptions should change.
You likely do not need dramatically more code authors. You may need:
- better test infrastructure
- one or two stronger developer productivity engineers
- clearer ownership maps
- tighter incident feedback loops
- more deliberate reviewer rotation for high-risk areas
This is the planning move most leadership teams miss.
If code generation gets cheaper and validation does not, the marginal dollar should go to systems that increase confidence per change. That might mean test runtime reduction, better ephemeral environments, or stronger observability, not more copilots.
PlanetScale’s public engineering content has often centered on safe schema change practices and operational correctness in database-heavy environments. That is the right instinct for AI-era productivity as well. In high-blast-radius systems, confidence infrastructure compounds more than generation infrastructure.
9. Redefine “done” for AI-assisted work
Most teams still define completion too early.
For AI-assisted changes, “done” should include:
- code merged
- tests meaningful, not just present
- observability added or updated
- rollback path documented
- ownership explicit
- post-deploy verification complete
If a generated feature lands quickly but creates a week of silent cleanup, you did not ship faster. You borrowed against future attention.
Datadog’s entire category exists because production truth matters more than development intent. The same principle applies here. AI can help produce intent faster. It cannot tell you whether reality stayed aligned unless your systems are instrumented to show it.
05 STRATEGIC TAKEAWAY
The winning move is not to maximize AI-generated code; it is to maximize trusted throughput. A CTO making budget and tooling decisions this quarter should assume that coding assistants will raise raw code production faster than the org’s ability to validate and absorb change. If you apply the framework above, you turn that imbalance into an advantage by investing in review scope reduction, test signal, architectural standardization, and flow metrics. If you do not, the likely outcome over the next two to three quarters is familiar: higher PR volume, slower review, noisier releases, and your most senior engineers spending their leverage on inspection instead of system design.
06 IMPLEMENTATION ANGLE
Start with a 60-day instrumentation sprint, not a company-wide mandate. Pick two teams with different profiles: one product team working in a well-tested service, and one team operating in a higher-risk system. Measure DORA metrics, review latency, reopen rate, flaky CI rate, and escaped defects before expanding licenses. This gives you a real baseline and exposes where your delivery system breaks under increased code volume.
Then create one paved-road package for a workflow where verification is cheap. A good first candidate is backend endpoint creation with built-in auth, tracing, logging, tests, and rollout defaults. If engineers can generate inside a constrained template, you get speed without multiplying review burden. If they generate into an unconstrained repo with weak contracts, you get activity masquerading as progress.
If your team is scaling quickly, this is also where a focused engineering-partner model helps. Amplify can help engineering teams scale by adding senior execution capacity around platform maturity, testing discipline, and systemized delivery workflows. The key is to use outside help to harden the system, not just increase output into the same bottlenecks.
