When AI writes implementation fast, engineering value shifts to framing, review, and risk ownership.
01 THE PROBLEM
Engineering judgment failure is the gap between code that works locally and systems that remain correct, operable, secure, and evolvable in production.
That gap is getting wider, not smaller.
AI coding tools compress implementation time from hours to minutes. They do not compress the need to make tradeoffs about correctness, rollback safety, failure isolation, security boundaries, data contracts, or long-term maintenance. The result is a predictable failure mode: teams mistake code generation speed for engineering throughput.
The consequence shows up within one or two quarters.
You ship more pull requests. Review queues get noisier. Test suites grow without becoming more meaningful. Incident load rises in subtle ways first: flaky jobs, retries masking bad assumptions, authorization checks copied inconsistently, schema drift, cost regressions, and “temporary” abstractions nobody wants to touch. By the time leadership notices, the problem is not that AI wrote bad code. The problem is that the team stopped evaluating whether the code represented good decisions.
This is the core measurement problem for CTOs and Staff+ engineers right now.
If a model can produce a plausible endpoint, migration, React component, Terraform module, or background worker on demand, what exactly are you evaluating in an engineer? Not typing speed. Not recall of syntax. Often not even implementation completeness. You are evaluating whether they can define constraints, detect hidden risk, reject seductive but wrong abstractions, and choose the version of “done” that fits the system.
That is engineering judgment.
It is also the scarcest capability in AI-assisted teams, because AI increases the volume of code that can be proposed without increasing the amount of contextual understanding inside the proposal.
The mistake most organizations make is operational. They update the toolchain before they update the evaluation model.
They approve copilots, coding agents, and PR automation. They leave career ladders, interview loops, code review norms, and delivery metrics mostly intact. That creates the wrong local incentive: generate more surface area, defer hard thinking, and rely on downstream review to catch what upstream judgment should have prevented.
DORA’s four key metrics — lead time for changes, deployment frequency, change failure rate, and time to restore service — matter here because they expose the real trade. Faster code generation should improve lead time and maybe deployment frequency. If change failure rate rises or mean time to recovery gets worse, you did not improve engineering performance. You shifted work from implementation to incident response. The DORA framework exists precisely to stop teams from confusing output volume with delivery quality.
In an AI-coding environment, that confusion becomes dangerous because the output volume can rise dramatically before the underlying system understanding changes at all.
The practical question is no longer, “Can the engineer code this?”
It is, “Can the engineer decide what should exist, what should not exist, what must be tested, and what risk the organization is actually accepting?”
That is what needs to be evaluated now.
02 WHY IT HAPPENS
This happens because AI is strongest in the layer of software work that is easiest to observe and weakest in the layer that matters most in production.
Implementation patterns are visible. Institutional context is not.
A model can infer how to write a queue consumer, a retry wrapper, or a database repository because the internet contains millions of examples. It cannot reliably infer your company’s blast radius constraints, undocumented migration scars, support burden, abuse patterns, compliance requirements, cost ceilings, or the exact failure mode that took down your billing flow nine months ago.
That missing context is where engineering judgment lives.
The structural problem is incentive misalignment.
Most organizations still reward engineering work through artifacts that AI can inflate: lines changed, tickets closed, visible velocity, PR count, or the appearance of autonomy. If a coding agent can turn a one-sentence ticket into 800 lines of plausible code by lunch, the org sees movement. What it does not automatically see is whether the engineer made the right decomposition, chose the right ownership boundary, preserved the existing invariants, or introduced a subtle class of operational risk.
The second root cause is that review systems were built for human-paced code creation.
Traditional code review assumes a bounded amount of surface area. An engineer writes something, another engineer reviews it, and both parties have enough context to reason about intent. AI breaks that assumption. Now one engineer can propose a week’s worth of code in an afternoon. Review quality does not scale linearly with generated volume. It collapses.
This is why experienced teams have long treated review size as a quality variable, not just a convenience variable. At Google, SmartBear, and others, longstanding engineering guidance has emphasized smaller changes because large diffs reduce review quality. That guidance matters more when AI can produce large, superficially coherent changesets on demand.
The third root cause is that organizations over-index on code correctness and under-index on system fitness.
ISO 25010 is useful here because it defines software quality beyond mere functional suitability: reliability, maintainability, security, performance efficiency, compatibility, and portability all matter. AI-generated code often looks acceptable on functional suitability. It is much more uneven on maintainability, analyzability, fault tolerance, and security posture. If your evaluation rubric ends at “tests pass,” you are grading the least interesting dimension.
The fourth root cause is organizational memory loss.
Judgment develops from exposure to consequences. That is why senior engineers improve after migrations gone wrong, capacity incidents, API design mistakes, and painful on-call rotations. AI can reduce exposure to the kinds of implementation details through which juniors used to build this intuition. If a newer engineer never has to wrestle manually with concurrency, schema evolution, backfills, auth edge cases, or flaky distributed workflows, they can produce more code while building less judgment.
That is not theoretical. It is a direct consequence of how skills compound. Will Larson has written extensively at StaffEng about engineering scope and decision-making as the scarce senior trait; coding is only one part of that role. The risk in AI-heavy environments is that organizations preserve the visible output while starving the apprenticeship path that creates future judgment.
The fifth root cause is misplaced confidence.
Large language models are excellent at producing coherent explanations for incorrect choices. That matters because software review is partly a persuasion process. A weak reviewer can be talked into a bad abstraction if the generated rationale sounds clean enough: “This pattern improves modularity,” “this hook centralizes state,” “this repository layer decouples storage,” “this retry logic increases resilience.” Every one of those statements can be true in the abstract and harmful in the actual codebase.
Charity Majors has spent years arguing that software decisions must be tied to operability and sociotechnical reality, not generic best practice. AI increases the need for that discipline because it mass-produces “best practice shaped” code detached from local constraints.
The final root cause is that leadership often tries to solve a judgment problem with a policy problem.
They ask for AI usage rules, approved tools, prompt templates, or mandatory disclosure. Those things may help with governance. They do not answer the hard question: how do you tell whether an engineer can use AI to produce fewer bad decisions, not just more code?
Until evaluation changes, behavior will not.
03 WHAT MOST GET WRONG
The most common misdiagnosis is thinking the problem is AI code quality.
It is not.
Bad code is a symptom. The actual problem is bad decision quality upstream and weak verification downstream. Teams focus on whether the model wrote elegant functions, idiomatic TypeScript, or comprehensive unit tests. Those are useful signals. They are nowhere near sufficient.
This leads to the first failed response: “We’ll just raise the code review bar.”
That fails because review is a lagging control. By the time a reviewer sees a large AI-generated diff, several expensive things have already happened: the solution shape has been chosen, the abstractions have spread, the engineer is anchored on the generated approach, and the reviewer is now asked to reverse-engineer intent from implementation. That is the most expensive moment to discover bad judgment.
The second failed response is over-reliance on tests.
Writing tests before implementation is good advice. In AI workflows, it is often excellent advice. But tests are only as strong as the engineer’s understanding of what matters. A shallow engineer with a fast model can produce a thoroughly tested wrong architecture.
This is the hidden trap in teams that treat test pass rate as a safety blanket. Tests verify expected behavior. They do not prove that you selected the right boundaries, avoided future coupling, chose the correct consistency model, or made the system debuggable at 3 a.m.
The third failed response is using AI fluency as a proxy for seniority.
The engineer who can prompt a model into producing a polished service scaffold is not necessarily the one you want deciding data ownership, idempotency semantics, rollout strategy, or whether the service should exist at all. Prompt quality matters, but mostly because it reveals the engineer’s prior judgment. It does not replace it.
The fourth failed response is doubling down on output metrics.
This is where things go off the rails fastest.
If managers praise ticket throughput, PR count, or “story points completed” after AI adoption, teams rationally optimize for generated volume. You end up with what looks like acceleration and feels like drag: more code to review, more behavior to observe, more edge cases to own, more long-tail maintenance, and more hidden inconsistency. DORA warned against simplistic productivity measurement long before AI. The warning matters more now.
The fifth failed response is treating every engineer as equally AI-enabled.
They are not.
A senior engineer with strong systems intuition often uses AI as a force multiplier. A mid-level engineer with partial understanding can use the same tools to generate more convincing mistakes. A junior engineer can become dependent on suggestions they cannot yet evaluate. The tool is constant. The judgment distribution is not.
A real-world analog exists in incident postmortems where automation amplified latent design issues rather than fixing them. Consider Knight Capital’s 2012 deployment failure, which the U.S. SEC documented after old code paths were unintentionally activated in production, causing roughly $460 million in losses in about 45 minutes. The lesson was not “automation is bad.” The lesson was that speed without robust controls, rollout discipline, and systems understanding is catastrophic. AI-assisted coding is not the same category of failure, but it follows the same pattern: compressed action, unchanged judgment, expensive consequences.
Another example comes from security. OWASP’s Top 10 has remained stubbornly relevant because the primary failure is rarely lack of access to secure code snippets. It is weak handling of trust boundaries, input validation, auth logic, secrets management, and dependency risk in real systems. AI can generate all the secure middleware boilerplate you want. If the engineer misunderstands authorization flow or threat modeling, the vulnerability still gets shipped.
What most teams get wrong, then, is where they place the control point.
They place it at code generation.
The control point should be at problem framing, change sizing, risk declaration, and evidence of understanding.
That is where judgment becomes visible.
04 THE FRAMEWORK
The approach that works is simple to state and harder to implement: evaluate engineers on decision quality before, during, and after code generation.
That requires a different operating model than “let the AI draft and the humans review.” You need explicit checkpoints where judgment is forced into the open.
Here is the framework.
1. Evaluate problem framing before any code is generated
Do not start by asking, “Can the engineer build it?”
Start with four written prompts the engineer must answer before implementation:
- What problem are we solving exactly?
- What constraints matter in this system?
- What failure modes are most likely or most expensive?
- Why is this approach better than the next two obvious alternatives?
This should fit on one page.
If an engineer cannot articulate these without code, AI will not save them. It will only accelerate a half-formed decision into a concrete artifact.
Stripe has written repeatedly about the importance of API design, clear invariants, and developer ergonomics in system evolution. That style of engineering discipline is relevant here: the work starts with the contract and the constraints, not the implementation flourish. In practice, teams that do this well force engineers to specify invariants first: idempotency expectations, latency budget, ownership boundary, rollback path, and migration shape.
A useful benchmark: if the change touches money movement, auth, data deletion, customer-visible state transitions, or cross-service contracts, require written framing before code review begins. No exceptions.
Tradeoff: this adds 10–30 minutes up front. It saves hours of review churn and weeks of hidden cleanup later.
2. Require test design before implementation design
This is where the freeCodeCamp-style practical advice intersects with senior engineering judgment: write or at least outline tests before asking AI for the implementation.
But make this stricter than generic TDD.
Require three classes of tests to be declared:
- Happy path
- Edge cases and invalid inputs
- System invariants or regression traps
For infrastructure or distributed changes, add:
- Rollback behavior
- Idempotency / retry safety
- Observability assertions
The point is not ritual purity. The point is to expose whether the engineer understands the real risk surface.
The Google SRE Book argues for designing around reliability targets and explicit failure handling rather than assuming correctness from intention. That maps directly here. If the engineer cannot define what should be observable and what should fail safely, they are not ready to delegate implementation to a model.
A practical threshold: if a generated PR adds more than 300 lines and has no explicit regression test plan, send it back before review. Large AI-generated changes without predeclared evidence requirements are where judgment debt accumulates.
Tradeoff: predeclared tests can feel heavyweight for trivial UI work. Fine. Scope it. Use it for medium- and high-risk changes, not every CSS tweak.
3. Grade architecture choices separately from code quality
This is the single biggest adjustment most teams need.
Split review into two explicit passes:
- Decision review
- Implementation review
Decision review asks:
- Should this logic live here?
- Is this service boundary correct?
- Are we introducing a new abstraction that will outlive its usefulness?
- Is the data ownership model clear?
- Is this change reversible?
Implementation review asks:
- Is the code correct?
- Is it readable?
- Are tests adequate?
- Are failure and logging behavior sane?
- Does it comply with security and style expectations?
If these are merged, implementation polish hides decision weakness.
GitHub’s engineering organization has written about maintaining developer velocity through internal platform improvements and disciplined workflows. One practical lesson from strong platform teams is that architecture decisions and implementation details should not be conflated. The former determine future speed; the latter mostly affect local quality.
An internal review template helps. Keep it short. Example headings:
- Chosen approach
- Rejected alternatives
- Blast radius
- Rollback plan
- Monitoring change
- Long-term maintenance cost
Tradeoff: separate review passes can feel slower. In practice they reduce late-cycle rewrite and reviewer fatigue.
4. Use production-facing metrics, not coding-facing metrics
If you want to know whether AI-assisted engineering is improving judgment, do not look first at generated LOC, prompt count, or acceptance rate of AI suggestions.
Look at outcomes the organization already trusts.
Use DORA’s four key metrics:
- Lead time for changes
- Deployment frequency
- Change failure rate
- Time to restore service
Then add three judgment-sensitive local metrics:
- Median PR size for production changes
- Rollback rate within 7 days of deploy
- Incidents linked to requirement or design misses, not coding bugs
This distinction matters. Coding bugs are implementation failures. Requirement and design misses are judgment failures.
A healthy pattern after AI adoption looks like this:
- Lead time decreases
- Deployment frequency stays flat or rises modestly
- Change failure rate stays flat or improves
- MTTR stays flat or improves
- PR size does not explode
- Rollback rate does not spike
If lead time improves by 20% but change failure rate rises sharply, you did not create leverage. You created rework.
The 2023 DORA research continues to reinforce that software delivery performance is multidimensional and that local efficiency metrics often distort system outcomes. This is exactly the mistake to avoid in AI rollouts.
Tradeoff: these metrics move more slowly than coding telemetry. That is good. Fast metrics are easy to game.
5. Constrain AI-generated change size by risk class
One reason AI degrades judgment visibility is that it increases code volume per unit time. You need mechanical limits.
A workable model:
Low-risk changes
- UI copy, internal tools, isolated tests, non-critical refactors
- AI can draft freely
- Single-reviewer approval acceptable
Medium-risk changes
- Internal APIs, migrations on non-critical tables, queue consumers, permission-adjacent features
- Require written framing
- Require declared test plan
- Cap initial PR size at roughly 400 changed lines, excluding generated snapshots or lockfiles
High-risk changes
- Auth, billing, data retention, public APIs, cross-service contracts, production infra, irreversible migrations
- Human-written design note required
- AI-generated code allowed only after decision review
- Feature flag or staged rollout mandatory
- Require observability and rollback plan
This is not bureaucracy. It is blast-radius pricing.
Cloudflare’s engineering culture gives useful examples here because the company operates at an internet edge where mistakes amplify fast. Their public engineering writing consistently shows disciplined rollout, protocol understanding, and operational controls. The lesson is not “copy Cloudflare’s process.” It is that the speed of shipping must be matched to the reversibility and scope of the change.
Tradeoff: strict gating can frustrate strong engineers on urgent work. That is preferable to normalizing silent high-risk generation.
6. Make reasoning auditable, not prompts performative
A lot of teams get stuck on whether engineers should paste their prompts into PRs.
That is the wrong artifact.
A prompt is not evidence of judgment. It is an interaction log.
What matters is whether the engineer can explain:
- what they asked the model to do,
- what they rejected,
- what they manually changed,
- and what they verified independently.
Ask for a short “AI assistance note” only when risk justifies it. Example:
- AI used for test scaffolding and initial repository wiring
- Rewrote transaction boundaries manually
- Added idempotency key handling manually
- Verified auth flows against existing service contract
- Rejected suggested retry loop because of duplicate side-effect risk
That note does two things. It improves review quality, and it trains engineers to distinguish delegation from abdication.
Mitchell Hashimoto has been explicit in public discussions about the importance of taste, tradeoffs, and deep systems understanding even as AI improves coding assistance. That is the right mental model: the value is not in hiding AI usage or documenting every keystroke. It is in making engineering reasoning inspectable.
Tradeoff: too much mandatory disclosure creates theater. Keep it focused on meaningful risk.
7. Train judgment through exposure to consequences
Judgment is not taught effectively through slide decks.
It is built through repeated contact with real decisions and their aftermath.
That means you need deliberate mechanisms:
- postmortem review tied back to originating design choices,
- shadowing in code reviews,
- pre-mortems for risky changes,
- migration retrospectives,
- on-call participation with senior debrief,
- architecture review that includes “what almost went wrong.”
Netflix’s engineering culture has long emphasized resilient systems, chaos thinking, and operational learning. Even if your stack is nothing like Netflix’s, the principle applies: teams build stronger judgment when they repeatedly examine failure modes as first-class engineering material, not as isolated incidents.
In AI-heavy teams, this matters more because the implementation path is less educational by default. You have to replace “I learned by writing all the plumbing myself” with “I learned by understanding exactly why this design survived production.”
Tradeoff: this consumes senior time. It is still cheaper than trying to hire judgment exclusively from outside.
8. Redefine seniority around decision ownership
Your ladder, interview process, and promotion packet must reflect the new bottleneck.
For senior and Staff+ roles, explicitly evaluate:
- quality of problem decomposition,
- ability to identify constraints early,
- system design under realistic tradeoffs,
- review quality on AI-assisted code,
- incident prevention, not just response,
- simplification decisions,
- evidence of maintaining team-wide standards under higher code volume.
Do not ask a Staff candidate only to implement a feature faster with AI. Ask them to reject an attractive but risky approach, redesign a generated solution, define rollout policy, or review an AI-written diff for hidden failure modes.
Will Larson’s work on Staff engineering roles is useful here because it frames seniority as leverage through direction, arbitration, and system shaping. AI increases the importance of exactly those behaviors.
Tradeoff: these qualities are harder to measure than output. That is why weak orgs avoid them. Strong orgs do not have that luxury.
9. Standardize the boring, reserve judgment for the sharp edges
The goal is not to hand-review every generated line with equal intensity. That does not scale.
Instead, codify the parts where judgment should not be repeatedly spent:
- service templates,
- auth middleware defaults,
- observability baselines,
- migration playbooks,
- test harnesses,
- secure dependency policies,
- CI quality gates,
- rollout checklists.
Then focus human attention where context matters:
- data model changes,
- customer-facing behavior,
- cross-team contracts,
- cost-impacting architecture,
- security boundaries,
- operational risk.
This is where companies like Shopify and Vercel have a clear lesson in public engineering output: platform leverage matters because it narrows the decision surface and makes the right path easier. AI works best when the paved road is good. It works worst when every generated change must rediscover your engineering standards from scratch.
Tradeoff: standardization can feel constraining to highly senior ICs. The payoff is fewer judgment cycles wasted on solved problems.
10. Audit whether AI is increasing maintenance load
Every quarter, run one brutally practical review:
- What percentage of AI-assisted changes were modified materially within 30 days?
- Which repos saw the sharpest increase in churn after merge?
- Which incidents traced back to generated assumptions?
- Which abstractions created in the last 90 days are already bypassed?
- Did support burden, on-call toil, or cloud cost rise in specific areas after adoption?
This is the check that prevents “faster now, worse later.”
Linear is a useful reference point because its public product and engineering reputation is built on restrained complexity and deliberate quality. The deeper lesson is not about any one toolchain. It is that speed only compounds when the codebase stays legible and maintainable. AI can either reinforce that discipline or destroy it.
Tradeoff: maintenance audits reveal uncomfortable truths about teams that appeared highly productive. Good. That is the point.
The Real Cost of Hiding Salary Ranges in Engineering Job Posts05 STRATEGIC TAKEAWAY
Engineering judgment must become a first-class management system, not an assumed trait. If you apply this shift, AI becomes a leverage layer on top of strong decision-making: lead time drops, senior review becomes higher signal, and your next 12 months of architecture are more coherent instead of more chaotic. If you do not, the bill arrives as change failure rate, maintenance drag, and expensive rework — exactly when a CTO needs the team shipping reliably through the next hiring plan, platform migration, or enterprise deal cycle.
06 IMPLEMENTATION ANGLE
Start with one team, not the whole org.
Pick a product squad that ships weekly and touches real production systems. For 30 days, require written framing for medium- and high-risk changes, separate decision review from implementation review, and track three metrics: median PR size, rollback rate within 7 days, and incidents caused by requirement or design misses. This is enough to expose whether AI is creating leverage or noise.
Then harden the paved road.
Add repo-level templates for design notes, test plans, rollback checklists, and AI assistance notes. Wire CI to enforce the basics automatically: linters, type checks, security scans, migration guards, and minimum observability hooks. The less judgment your team spends on repeatable hygiene, the more they can spend on architecture, failure modes, and customer impact.
If you are scaling from 30 to 100 engineers, this becomes an org design issue as much as a tooling issue. The teams that stay fast are the ones that codify judgment into review structures, platform defaults, and promotion criteria early. That is one area where Amplify can help engineering teams scale: not by replacing technical leadership, but by helping operationalize the systems around hiring, calibration, and team growth that keep decision quality from fragmenting as code generation gets cheaper.
