Ask an agent to fix a bug and add tests. It will do both, and do them well, in one pass. The tests will pass. Coverage will go up. The PR will look excellent.
Now notice what just happened. The same process produced the code and the specification it is checked against. If the agent misread the requirement, it misread it the same way twice: the code does the wrong thing, and the test says the wrong thing is correct. Both artifacts agree. Nothing in your pipeline can tell.
You didn’t get verification. You got a system that documented its own interpretation and then confirmed it.
We have always known this was a problem. It’s why “write your own tests” always came with a quiet asterisk, and why real QA organizations kept some independence. But it was a small problem, because writing tests was expensive. People wrote few of them and thought hard about each one. The cost forced the discipline.
Generating ten thousand tests now costs less than a coffee.
Coverage percentage measures how much of your code is exercised by test code. It has never measured how much of your intended behavior is proven. When writing tests was expensive, those two things moved together. They don’t anymore. The metric outlived the link that made it work.
This is part five of Rethinking the SDLC. The previous piece routed changes by risk and demanded machine evidence at the gate. This one is about what belongs in that evidence, because “we have 87% coverage” is not an answer.
The number was always weak. Now it’s actively misleading.#
Coverage was a stand-in, and a fair one under one assumption: that a human wrote each test because they were thinking about a way the code could be wrong. The number never measured quality. It counted things produced by an act of thought. The thought was the value. The count was the receipt.
Take the cost away, and you can print receipts without the thought. Worse, you print them at exactly the moment the thought was most needed and least present, because the thing writing them just finished writing the code and has already committed to one reading of the ticket.
There’s a name for the resulting artifact class: tests that cannot fail. A test that exercises a line but asserts nothing meaningful. A test that mocks the dependency whose interaction was the actual risk. A test that asserts the implementation’s current output rather than the required behavior — snapshot tests are a whole industry of this. A test whose setup is so specific that it can only pass. Each of these raises coverage and catches nothing, ever.
And the evidence says this is not a thought experiment. The detail I’d point to in Cognition’s FrontierCode benchmark is a design decision rather than a score: when they built a benchmark to predict whether a maintainer would merge an agent’s PR, test quality had to become its own graded dimension. A passing suite was not a good enough stand-in — they had to judge the tests separately from whether the tests passed. That’s the whole problem of this article, discovered independently by people building a measurement instrument. (For the record, every frontier model still fails more than half the hard tasks.) Veracode found risky security flaws in 45% of tasks across more than 100 models. Georgetown’s CSET found that nearly half of AI-generated snippets held bugs, and many of them mattered. A January 2026 study of autonomous coding agents found cognitive complexity up 39% and static-analysis warnings up 18%.
All of that shipped through pipelines with green test suites. The suites weren’t lying. They were answering a question nobody should have been asking.
The only question that matters: who wrote the oracle?#
Strip testing down to its moving parts and there are two. There is the execution — run the code with some inputs. And there is the oracle — the thing that decides whether the result was correct.
Execution is cheap and always was. The oracle is the entire value.
So the question that decides whether a test is worth anything is not “does it pass” or “what does it cover.” It is: where did the oracle come from, and did it come from somewhere other than the code it checks?
Run your existing suite through that question and it sorts itself immediately:
- Oracle from the requirement, written before the code → real verification.
- Oracle from a mathematical property of the domain → real verification, and stronger, because it holds for inputs nobody listed.
- Oracle from behavior seen in production → real verification for regressions.
- Oracle from a second, separate implementation → real verification, and expensive.
- Oracle from what the code happens to do today → not verification. A change detector. Useful for refactoring, worthless for correctness.
- Oracle from the same agent, in the same pass, from the same reading of the ticket → not verification. It only checks that one reading agrees with itself.
That last line is the default state of most agent-generated test suites in production right now. Not because anyone chose it. Because it’s what “add tests” produces.
Anthropic’s engineering team put the conclusion plainly while describing what it took to run 16 parallel agents at a single hard goal: “it’s important that the task verifier is nearly perfect, otherwise Claude will solve the wrong problem.” Not the model. The verifier. When execution has no limit, the verifier is the only thing standing between you and confident, wrong output at scale. And if the verifier came from the same place as the output, you have no verifier.
Verification coverage: the metric that replaces the metric#
Here’s the definition I use, and it’s deliberately harder to hit than the number it replaces.
Verification coverage: the share of your system’s stated required behaviors that are proven by an oracle that did not come from the code.
Three properties make it useful.
It counts behaviors, not lines. The bottom half of the fraction comes from your intent packages — the outcomes you said must be true. So the metric can’t be gamed by generating more test code — and you can’t calculate it at all if you never wrote your intent down. That’s a feature. It puts pressure on the upstream artifact.
It requires independence, and you can audit that. For any behavior you can ask where its oracle came from, and the answer is a fact about your process, not an opinion.
It can go down when the system grows. Ship a new behavior with no independent proof and verification coverage drops. Line coverage would go up. That flip is the whole point: the metric now moves with your real risk.
You will not get to 100% and shouldn’t try. Some behaviors genuinely can’t be proven mechanically — “the error message is helpful,” “this abstraction will make sense to the next team.” Those go on a written list of human-owned behaviors, and that list is valuable on its own. It maps where human judgment is load-bearing, and it tells you exactly where you can never fully automate the gate.
The proof-strength ladder#
Not all evidence is equal. Rank it, then require a rung by change class.
Rung 1 — Example-based tests written against the requirement. The ordinary unit test, with one condition: the assertion traces back to a stated intent, not to whatever the code printed. Written before the implementation, or by something that hasn’t seen it. Weakest real proof. Fine for most Class A and B work.
Rung 2 — Property-based tests. These check invariants — things that must stay true — across generated inputs, instead of checking one output for one input. “Serializing then deserializing returns the original.” “Applying this twice equals applying it once.” “The total never goes negative.”
This is the most underused high-leverage technique in the industry right now, and agents changed the cost of it completely. Property tests used to be hard to write. You had to think about the domain in the abstract, then fight a generator library. That was a real barrier. It isn’t one now: an agent will happily draft twenty candidate properties for a module, and your job becomes deciding which ones are actually true. Reviewing twenty proposed invariants is an excellent use of senior attention. Writing four hundred example tests never was.
Rung 3 — Mutation testing. Break the code on purpose and check that the suite notices. This is the direct answer to “are these tests real,” and it is the only technique that measures the suite instead of the code.
Mutation score should replace coverage percentage as your headline test-health metric, and it is affordable now in a way it wasn’t five years ago, because compute is cheap and a machine can do the analysis. If your suite has 90% coverage and a 30% mutation score, you have a suite that runs and doesn’t check. That is worth knowing before an incident tells you.
Rung 4 — Production-derived verification. Real traffic shapes, replayed. Recorded request/response pairs from production as regression fixtures. Shadow traffic against the new path with output comparison. Differential testing of old versus new implementation on real inputs.
The oracle here is reality, which makes it the strongest everyday proof you can get. It is also the one that needs your observability to be structured enough to feed it — which is why the observability work later in this series matters more than it sounds.
Rung 5 — Formal and semi-formal proof. Type-level guarantees, checks that every case is covered, model checking on state machines and concurrency, invariant proofs on critical algorithms. Expensive, narrow, and correct. Keep it for the handful of places where being wrong is not an option: money movement, auth decisions, data deletion, distributed consensus.
Then the rule that puts the ladder to work: the rung you need is a property of the change class — declared once, enforced by the pipeline.
Independence is an engineering problem, and it’s solvable#
“Have a different agent write the tests” points the right way, but it is not enough on its own. Four mechanisms make independence real.
Verification-first ordering. The verification section of the intent package is written and reviewed before implementation starts. The oracle is derived from the requirement because it existed before the code did. This is old-fashioned TDD, and the reason to bring it back isn’t discipline. It is the cheapest source of independence you have.
Separate context, separate lane. The verifying agent gets the intent package, the interfaces, the invariants, and the production behavior. It does not get the implementation diff. It cannot copy the shape of what was built, which is exactly the failure you are trying to prevent.
Give it a hostile goal. Its instruction is to break the change, not to cover it. “Find inputs where the stated intent is violated.” Tests that pass first time are the least interesting output of that process.
Mutation as the meta-gate. Whatever generated the tests, mutation testing audits them. This is the check that doesn’t care about your process story. It answers, with evidence, whether the suite can spot wrong code. Set a threshold, enforce it, and every other kind of test theater becomes visible.
Note the pattern across all four: independence isn’t a matter of trusting a different model. It’s a matter of controlling what the verifier is allowed to know.
Six moves#
- Run mutation testing on your three highest-consequence modules this week. You will get a number that changes every conversation about test quality on your team. Expect it to be bad. That’s the finding.
- Demote coverage from a gate to a warning light. Keep measuring it — a sharp drop still means something. Stop making it a merge requirement and stop putting it on a slide.
- Write verification requirements per change class, once. Which rung is required for notification delivery, for auth, for migrations, for the UI. This document is a bigger contribution to reliability than any individual test anyone will write this year.
- Split the lanes. Implementation agent and verification agent, different contexts, verification spec first. Making this the default in your harness costs a day, and it changes everything downstream.
- Have an agent propose properties for one core module. Ask for twenty candidate invariants. Sit with a senior engineer and mark which are actually true. You’ll find at least one you believed and one you were wrong about — and that second one is a bug you hadn’t found yet.
- Build one production-replay harness. One path, real recorded traffic, differential comparison. It will catch a kind of regression your unit tests can never see, by design, and it is the base for everything in the CI/CD piece.
Verified means something. Say what.#
Here’s the sentence I’d make every engineering organization write down and put next to its definition of done:
A behavior is verified when an oracle that did not come from the implementation says it is correct, and we can show you the oracle.
Everything else — coverage percentages, green suites, passing pipelines, PR approvals — is evidence about verification, not verification. All of it can be produced without any verification happening at all — and now it can be produced for free, which means it will be.
This is the part of the inversion I think we discuss least. Cheap execution didn’t only make code cheap. It made the appearance of correctness cheap — and that is far more dangerous than obviously bad code, because obviously bad code gets caught.
The craft here is old and unglamorous. Knowing what must be true. Knowing how you would find out you were wrong. Not trusting a green check. That is testing, and it was never really about writing tests.
So: pull up your highest-coverage module. Break something in it on purpose — flip a comparison, drop a boundary check, off-by-one an index. Then run the suite.
If it stays green, you now know exactly what your coverage number was worth.
I lead AI transformation for a global SaaS platform, and the verification lanes described here are what my agents actually merge through. If you’ve moved off coverage as a gate, or you’re running mutation testing at scale, I want to hear what it cost and what it caught. Find me on X, LinkedIn, or Telegram.