DORA is the best research program our industry has produced, and this article is not a dunk on it. I’ve written admiringly about applying the 2025 findings in real life, and their central conclusion — that AI amplifies whatever foundation you give it — is the most useful sentence anyone published about this transition.
But here is the thing about a great stand-in metric. It works because of something true about the world, not because of anything in the metric itself. And when that thing stops being true, the metric does not tell you. It keeps producing numbers.
Deployment frequency worked because deploying frequently required you to have solved a long list of hard problems: automated testing, trunk-based development, small batches, feature flags, fast rollback, low coupling. You could not fake it. A team deploying fifty times a day had genuinely built something, and the metric was a short summary of all of it.
Lead time for changes worked for the same reason. Getting from commit to production quickly meant your pipeline, your review, and your architecture were all in decent shape.
Both of those metrics measured how much friction you had taken out of a scarce, expensive, human-typed change. That was the constraint. Measuring how fast a scarce thing moved through your system told you almost everything about the system.
Now change is not scarce. And the metrics still produce numbers.
Deployment frequency and lead time became inflatable the moment change generation stopped being the constraint. A metric you can inflate without improving anything is not a KPI. It is a vanity metric with an impressive past.
This is part eleven of Rethinking the SDLC. Everything before this built mechanisms. This one is about what you put on the wall.
The specific way each key breaks#
Let me be exact rather than sweeping, because two of the four keys survive in a changed form.
Deployment frequency: now an input metric.
Point three agent lanes at your backlog and your deploy count goes up next week. Nothing about your capability changed. You added generation capacity, and deployment frequency counted it.
Worse, you can now game it with things that look like best practice. Split changes smaller — deploy count up. Let agents open more PRs — up. Batch less — up. Every one of those is reasonable engineering advice, and you can follow all of them without one extra customer getting something that works.
That is the part people resist, so let me say it plainly. A metric does not become safe because the research behind it is good. Deployment frequency earned its authority under conditions that no longer hold, and Goodhart’s law — as soon as a measure becomes a target, it stops being a good measure — has never cared where a metric came from. Amazon’s deleted AI-usage leaderboard is the same warning with a shorter fuse. A respected metric pointed at a constraint that moved fails the same way, just slower, and with more trust behind it.
Lead time for changes: now unclear about whose time it counts.
Lead time from commit to production is measuring the wrong span, because commit is no longer where the work starts. If an agent produces a commit four minutes after being pointed at a ticket, your lead time looks superb — and it leaves out the two weeks the ticket sat half-written and the three agent attempts that were thrown away. The four days the PR then waited for a reviewer are inside the span, which is the other half of the problem: that queue is now the bulk of the number, and a metric named for change is mostly reporting how busy your reviewers were.
This is exactly the gap that shows up in the telemetry. LinearB data across 8.1 million PRs in 4,800 organizations found developers feeling 20% faster while merged-to-production time ran 19% slower. METR’s randomized trial with experienced open-source developers found them about 19% slower with AI while believing they were 20% faster. Same direction, same size, two completely different research methods. That is not noise. That is a measurement line drawn in the wrong place.
Change failure rate: survives, and gets more important.
This one holds up. It measures outcomes: things that reached production and went wrong. Agents cannot inflate it by producing more. Producing more only makes the bottom of the fraction bigger.
One condition: the definition has to include reverts and follow-up fixes, not just declared incidents. A change that shipped, quietly broke something, and got patched twice was a failed change. If your change failure rate looks suspiciously low while your revert rate climbs, you are counting incident declarations, not failures.
Time to restore service: survives, but needs splitting.
MTTR still means something, and it means much more once you break it apart, because agents change the pieces very unevenly. Split it into detect, get your bearings, decide, fix, verify. Agents almost wipe out getting your bearings and can shorten fix. They do nothing for decide, which is human by design. One combined MTTR number hides which part you actually improved — and, more dangerously, hides the decide phase getting longer because the humans are overloaded.
So: two of four still work. The two that broke are the two everyone puts on slides.
The measurement problem underneath all of it#
There is a deeper problem than any single metric, and it is why patching DORA is not enough.
Every metric in the old set counted units of change: deploys, commits, PRs, tickets. That was a fair thing to count while producing a unit of change took human effort, because the unit tracked the work. It stood in for effort spent, and effort was the scarce thing.
Units of change now track nothing. An agent can produce forty of them from the same input a human needed for one. Any metric built on units of change now measures how much you can generate — which stretches on demand, can be bought, and is not a capability.
So you have to count different things. Verified outcomes. Human decisions. Intents delivered. Behaviors proven. Those are still scarce, and scarce things are the right things to count.
Uber’s COO gave the cleanest illustration of what happens when you don’t do this. The company reported 95% engineer AI adoption and admitted the connection from that to product outcomes “is not there yet.” Sit with that. An adoption number can hit 95% while nobody has shown any link to results. It never measured value. It measured belief.
Seven signals for an agentic production system#
Here is the replacement set. For each one: what it is, how to work it out, and how people will game it — because a metric you cannot game is a metric you have not thought about hard enough.
1. Intent-to-verified-behavior time (IVB)#
What: the time from an intent being accepted as ready to the behavior being provably correct in production. Not commit to deploy. Intent to proven.
Why it leads: it stretches across every constraint in the new system. Vague intent lengthens it. Stale context lengthens it. Weak verification lengthens it. Review queues lengthen it. Rework lengthens it. You cannot improve IVB by generating more — which is exactly what deployment frequency lost.
How to work it out: stamp the time when the intent package passes readiness, and again when verification confirms the behavior in production. Report the spread, not the average — the slow tail is where your organization’s real problems live. Publish it weekly where every engineer and every executive can see it.
How it gets gamed: by loosening the readiness rules so intents “start” later, or by weakening what counts as verified. You can see both if you also track verification coverage, which is why these two travel together.
2. Verification coverage#
What: the share of stated required behaviors proven by an oracle that did not come from the code itself. Defined properly in the proof-of-behavior piece.
Why: it is the best single predictor of whether you can safely raise autonomy. It is also the only metric here that gets worse when you ship unverified behavior, while line coverage gets better.
How it gets gamed: by not writing behaviors down, which shrinks what you are dividing by. Guard it with a mutation score on the critical paths — a check with evidence behind it, which does not care about your process story.
3. Human intervention rate, per workflow#
What: for each agent workflow, the share of runs where a human had to step in, plus how often humans override the recommendations.
Why: this is your judgment budget, measured. It shows where autonomy is real and where it is only a hope. And the second half — override rate — is the approval-theater detector. A workflow where humans approve 99.8% of recommendations does not have a human in the loop. It has a human next to the loop, clicking.
How it gets gamed: by relabelling those moments as normal operation. Define stepping in once, mechanically: a human changed, rejected, or corrected the agent’s output.
4. Rework rate#
What: the share of merged changes that get reverted, fixed again, or meaningfully changed within 30 days.
Why: rework is the true cost of unverified output, and it is where the 180% more code shipping only 30% more software gap actually lives. High deploy frequency with high rework is not speed. It is a system doing the same work over and over and counting every attempt.
How it gets gamed: by shipping follow-up fixes that never mention the original. Link them mechanically, by which files they touch and when, rather than trusting commit messages.
5. Cost per verified outcome#
What: the full cost of each verified outcome — per bug investigated, per verified change merged, per incident resolved — with failed attempts charged to the same total, and compared against the true cost of an engineer’s time. Managing agent spend deserves its own article. What matters here is the job this number does inside this set.
Why it belongs in a metric set and not just a budget review: it is the only signal measured in money, which makes it the one that can settle arguments between the other six. IVB says a workflow is slow. Verification coverage says it is under-proven. Intervention rate says it needs humans. None of those tell you whether to fix it, fund it, or switch it off. Cost per verified outcome does, because it puts every workflow on the same scale as the alternative — a person doing it.
How it gets gamed: by defining outcomes loosely, which is why this signal is only as trustworthy as signal 2. “Verified” has to mean a gate or a named human confirmed the result, not that a PR opened. Loosen the definition and this number improves while nothing else does. That is the most common way a cost metric lies.
6. Prevented-incident rate#
What: defects caught by each gate before they reach production, credited to the layer that caught them: machine floor, adjudication, human review, staged verification.
Why: this is the only metric that makes money spent on verification visible. Without it, a gate that works looks like pure cost. It produces no incidents, so it looks like it does nothing. With it, you can say “the scope check caught 34 out-of-scope changes last quarter, four of which touched auth.” That is a sentence someone will fund.
It is also how you earn the right to raise autonomy. Gate-catch rates plus sampled-audit results are the evidence that moves a workflow up a tier.
How it gets gamed: by counting trivial catches. Weight them by how serious they were, not by how many there were.
7. Comprehension coverage#
What: for each critical subsystem, whether at least two people can explain how it works and why it is built that way. Checked every quarter, by conversation, not by survey.
Why: it is the only early warning of whether your organization can still make good architectural decisions, and it is the one kind of debt that damages your ability to judge all the others. It is rough and it is a matter of opinion, and it is still worth more than any static-analysis dashboard — because a zero here means an organization has lost ownership of a system it depends on.
How it gets gamed: by asking people to self-report. Have someone else assess it, in a conversation, with follow-up questions.
Retire these#
Let’s say the quiet part out loud. These should come off your dashboards for good:
- Lines of code, by anyone, ever. It was always a cost, never an output. Now it explodes — it counts raw material used up, not anything produced.
- PR count and commit count. Stretchy, easy to inflate, and they tell you nothing.
- AI adoption percentage. A comfort blanket — see above. It measures belief, not value.
- Tokens used, and per-engineer spend caps. Both measure inputs, so both can be gamed, and neither tells you what the money bought. Use cost per verified outcome instead, and keep the caps only as a circuit breaker against a runaway.
- Test coverage as a gate. Keep it as a warning light — a sharp drop still means something. Stop making it a condition of merging.
- Story points and velocity, in agent-heavy lanes. Both were tools for handing out scarce human capacity. That is not what is scarce now.
If a metric goes up when you add generation capacity without improving anything, it’s measuring the wrong constraint. That’s the whole test, and it takes ten seconds per metric.
The DORA finding that actually matters most#
Here is the twist worth sitting with. DORA’s own recent work holds the most important number in this whole debate, and it is not one of the four keys.
Their ROI model projects roughly $11.6M in first-year returns on $8.4M invested for a 500-person organization — 39% ROI, about an eight-month payback. Then it attaches a condition: that return depends on the organization already having a strong platform and strong CI. Without those, the model does not predict a smaller return. It predicts things get worse before they get better.
Everyone quotes the number. The condition is the finding.
Strong platforms and CI are verification capacity. What DORA measured is that agent generation converts to shipped value only in organizations that can verify at speed, and it converts to expensive inventory in organizations that can’t. That is the amplifier argument stated in dollars: same tools, opposite outcomes, and the thing that decides which one you get is a capability no four-key dashboard shows.
Which is exactly why the metric set has to change. If verification capacity decides whether AI pays off, and no metric on your wall measures verification capacity, then your wall cannot tell you whether your biggest engineering investment is working.
Four moves#
- Start measuring IVB this quarter, even roughly. Two timestamps and a weekly spread. Publish it. It will start more useful arguments than everything else on this list put together.
- Add rework rate and human intervention rate next to your existing DORA board. Don’t remove DORA — put the new numbers beside it and let the comparison do the work. When deploy frequency is up 60% and rework is up 80%, nobody needs a memo.
- Compute cost per verified outcome for three workflows. Pick one you love, one you suspect, and one nobody’s examined. Charge failures to the workflow. Then feed the winners, kill the losers, and let the accounting be boring.
- Start crediting prevented incidents to the gate that caught them. It is the only way money spent on verification ever becomes visible enough to fund, and it is what earns you the autonomy increases you actually want.
Measure what’s scarce#
The principle is one line, and it survives whatever comes after agents.
Measure what’s scarce. For fifty years that was human execution capacity, and DORA measured its friction better than anyone. Today the scarce things are clear intent, accurate context, real verification, sound judgment, and effective control. Every signal above counts one of those.
Reality is patient. It wins every argument it is allowed to join — but only if you let it into the room, and a dashboard full of inflatable numbers is how you keep it outside.
So one question, and it’s the one I’d want answered before any planning cycle. Intent to verified behavior in production, this month versus six months ago: do you know your number?
Not your deploy count. Not your adoption rate. Not how fast the team feels. The number.
If you don’t have it, you are planning against a feeling. And a feeling with a budget attached is no longer a feeling. It is a risk you own.
I lead AI transformation for a global SaaS platform, and IVB plus cost per verified outcome are the two numbers I actually steer by. If you’ve instrumented either — or you think DORA’s four keys hold up better than I’m giving them credit for — come argue. Find me on X, LinkedIn, or Telegram.