Ask an agent to fix a bug and add tests. It will do both, competently, in one pass. The tests will pass. Coverage will go up. The PR will look excellent.
Now notice what just happened. The same process produced the implementation and the specification it’s checked against. If the agent misunderstood the requirement, it misunderstood it consistently — the code does the wrong thing and the test asserts that the wrong thing is correct. Both artifacts agree. Nothing in your pipeline can tell.
You didn’t get verification. You got a system that documented its own interpretation and then confirmed it.
We have always known this was a problem. It’s why “write your own tests” has an implicit asterisk on it, and why real QA organizations kept some independence. But it was a small problem, because writing tests was expensive, so people wrote few of them and thought hard about each one. Cost imposed discipline.
Generating ten thousand tests now costs less than a coffee.
Coverage percentage measures how much of your code is exercised by test code. It has never measured how much of your intended behavior is proven. When test generation was expensive, those two things correlated. They don’t anymore, and the metric survived the correlation dying.
This is part five of Rethinking the SDLC. The previous piece routed changes by risk and demanded machine evidence at the gate. This one is about what belongs in that evidence, because “we have 87% coverage” is not an answer.
The number was always weak. Now it’s actively misleading.#
Coverage was a proxy, and it was a reasonable one under a specific assumption: that a human wrote each test because they were thinking about a way the code could be wrong. The number wasn’t measuring quality — it was counting artifacts produced by an act of thought. The thought was the value. The count was the receipt.
Remove the cost, and you can produce receipts without the thought. Worse, you can produce them at exactly the moment the thought was most needed and least present, because the thing generating them just finished writing the code and has already committed to an interpretation.
There’s a name for the resulting artifact class: tests that cannot fail. A test that exercises a line but asserts nothing meaningful. A test that mocks the dependency whose interaction was the actual risk. A test that asserts the implementation’s current output rather than the required behavior — snapshot tests are a whole industry of this. A test whose setup is so specific that it can only pass. Each of these increments coverage and detects nothing, forever.
And the evidence says this is not hypothetical. The detail I’d point to in Cognition’s FrontierCode benchmark is a design decision rather than a score: when they built a benchmark to predict whether a maintainer would merge an agent’s PR, test quality had to become its own graded dimension. A passing suite wasn’t a usable proxy — they had to assess the tests separately from whether the tests passed. That’s the whole problem of this article, discovered independently by people building a measurement instrument. (For the record, every frontier model still fails more than half the hard tasks.) Veracode found risky security flaws in 45% of tasks across more than 100 models. Georgetown’s CSET found that nearly half of AI-generated snippets contained bugs that were often impactful. A January 2026 study of autonomous coding agents found cognitive complexity up 39% and static-analysis warnings up 18%.
All of that shipped through pipelines with green test suites. The suites weren’t lying. They were answering a question nobody should have been asking.
The only question that matters: who wrote the oracle?#
Strip testing down to its mechanics and there are two parts. There’s the execution — run the code with some inputs. And there’s the oracle — the thing that decides whether the result was correct.
Execution is cheap and always was. The oracle is the entire value.
So the question that determines whether a test is worth anything is not “does it pass” or “what does it cover.” It’s: where did the oracle come from, and was it derived independently of the implementation?
Run your existing suite through that question and it sorts itself immediately:
- Oracle derived from the requirement, before implementation → real verification.
- Oracle derived from a mathematical property of the domain → real verification, and stronger, because it holds across inputs nobody enumerated.
- Oracle derived from observed production behavior → real verification for regressions.
- Oracle derived from a second independent implementation → real verification, and expensive.
- Oracle derived from what the code currently does → not verification. A change detector. Useful for refactoring, worthless for correctness.
- Oracle derived by the same agent, in the same pass, from the same reading of the ticket → not verification. A consistency check on one interpretation.
That last line is the default state of most agent-generated test suites in production right now. Not because anyone chose it. Because it’s what “add tests” produces.
Anthropic’s engineering team put the conclusion plainly while describing what it took to run 16 parallel agents at a single hard goal: “it’s important that the task verifier is nearly perfect, otherwise Claude will solve the wrong problem.” Not the model. The verifier. When execution is unbounded, the verifier is the only thing standing between you and confidently-wrong output at scale — and if the verifier came from the same place as the output, you have no verifier.
Verification coverage: the metric that replaces the metric#
Here’s the definition I use, and it’s deliberately harder to hit than the number it replaces.
Verification coverage: the fraction of your system’s stated required behaviors that are proven by an oracle derived independently of the implementation.
Three properties make it useful.
It’s denominated in behaviors, not lines. The denominator comes from your intent packages — the outcomes you said must be true. This means the metric can’t be gamed by generating more test code, and it also means you can’t compute it if you never wrote your intent down. That’s a feature. It puts pressure on the upstream artifact.
It requires independence, which is auditable. For any given behavior you can ask where its oracle came from, and the answer is a fact about your process, not an opinion.
It can go down when the system grows. Ship a new behavior with no independent proof and verification coverage drops. Line coverage would go up. That inversion is the entire point: the metric now moves in the direction of your actual risk.
You will not get to 100% and shouldn’t try. Some behaviors genuinely can’t be proven mechanically — “the error message is helpful,” “this abstraction will make sense to the next team.” Those go on an explicit list of human-owned behaviors, which is itself valuable: it’s the map of where human judgment is load-bearing, and it tells you exactly where you can never fully automate the gate.
The proof-strength ladder#
Not all evidence is equal. Rank it, then require a rung by change class.
Rung 1 — Example-based tests written against the requirement. The ordinary unit test, with one condition: the assertion traces to a stated intent, not to observed output. Written before the implementation, or by something that hasn’t seen it. Weakest real proof. Fine for most Class A and B work.
Rung 2 — Property-based tests. Assert invariants across generated inputs instead of asserting specific outputs for specific inputs. “Serializing then deserializing returns the original.” “Applying this twice equals applying it once.” “The total never goes negative.”
This is the highest-leverage underused technique in the industry right now, and agents changed its economics completely. Property tests were historically hard to write — you had to think abstractly about the domain and then wrestle a generator library. That was a real barrier. It isn’t one anymore: an agent will happily draft twenty candidate properties for a module, and your job becomes reviewing which ones are actually true. Reviewing twenty proposed invariants is a superb use of senior attention. Writing four hundred example tests never was.
Rung 3 — Mutation testing. Deliberately break the implementation and check that the suite notices. This is the direct answer to “are these tests real,” and it’s the only technique that measures the suite rather than the code.
Mutation score should replace coverage percentage as your headline test-health metric, and it is now affordable in a way it wasn’t five years ago because the compute is cheap and the analysis is automatable. If your suite has 90% coverage and a 30% mutation score, you have a suite that runs and doesn’t check. That is worth knowing before an incident tells you.
Rung 4 — Production-derived verification. Real traffic shapes, replayed. Recorded request/response pairs from production as regression fixtures. Shadow traffic against the new path with output comparison. Differential testing of old versus new implementation on real inputs.
The oracle here is reality, which makes it the strongest routine proof available. It’s also the one that requires your observability to be structured enough to feed it — which is why the observability work later in this series matters more than it sounds.
Rung 5 — Formal and semi-formal proof. Type-level guarantees, exhaustiveness, model checking on state machines and concurrency, invariant proofs on critical algorithms. Expensive, narrow, and correct. Reserve it for the handful of places where being wrong is unacceptable: money movement, auth decisions, data deletion, distributed consensus.
Then the rule that makes the ladder operational: the required rung is a property of the change class, declared once, enforced by the pipeline.
Independence is an engineering problem, and it’s solvable#
“Have a different agent write the tests” is directionally right and insufficient on its own. Four mechanisms make independence real.
Verification-first ordering. The verification section of the intent package is written and reviewed before implementation starts. The oracle is derived from the requirement because it existed before the code did. This is old-fashioned TDD, and the reason to revive it isn’t discipline — it’s that it’s the cheapest available source of independence.
Separate context, separate lane. The verifying agent gets the intent package, the interfaces, the invariants, and the production behavior. It does not get the implementation diff. It cannot pattern-match to what was built, which is exactly the failure you’re preventing.
Adversarial objective. Its instruction is to break the change, not to cover it. “Find inputs where the stated intent is violated.” Tests that pass first time are the least interesting output of that process.
Mutation as the meta-gate. Whatever generated the tests, mutation testing audits them. This is the check that doesn’t care about your process story — it empirically answers whether the suite can detect a wrong implementation. Put a threshold on it and enforce it, and every other form of test theater becomes visible.
Note the pattern across all four: independence isn’t a matter of trusting a different model. It’s a matter of controlling what the verifier is allowed to know.
Six moves#
- Run mutation testing on your three highest-consequence modules this week. You will get a number that reframes every conversation about test quality on your team. Expect it to be bad. That’s the finding.
- Demote coverage from a gate to a diagnostic. Keep measuring it — a sharp drop still means something. Stop making it a merge requirement and stop putting it on a slide.
- Write verification requirements per change class, once. Which rung is required for notification delivery, for auth, for migrations, for the UI. This document is a bigger contribution to reliability than any individual test anyone will write this year.
- Split the lanes. Implementation agent and verification agent, different contexts, verification spec first. Making this the default in your harness costs a day and changes the character of everything downstream.
- Have an agent propose properties for one core module. Ask for twenty candidate invariants. Sit with a senior engineer and mark which are actually true. You’ll find at least one you believed and one you were wrong about — and that second one is a bug you hadn’t found yet.
- Build one production-replay harness. One path, real recorded traffic, differential comparison. It’ll catch a class of regression your unit tests structurally cannot see, and it’s the foundation for everything in the CI/CD piece.
Verified means something. Say what.#
Here’s the sentence I’d make every engineering organization write down and put next to its definition of done:
A behavior is verified when an oracle that did not come from the implementation says it is correct, and we can show you the oracle.
Everything else — coverage percentages, green suites, passing pipelines, PR approvals — is evidence about verification, not verification. Most of it can be produced without any verification occurring at all, and now it can be produced for free, which means it will be.
This is the part of the inversion I find most under-discussed. Cheap execution didn’t just make code abundant. It made the appearance of correctness abundant, and the appearance of correctness is far more dangerous than obviously bad code, because obviously bad code gets caught.
The craft here is old and unglamorous. Knowing what must be true. Knowing how you’d find out you were wrong. Being suspicious of a green check. That’s testing, and it was never really about writing tests.
So: pull up your highest-coverage module. Break something in it on purpose — flip a comparison, drop a boundary check, off-by-one an index. Then run the suite.
If it stays green, you now know exactly what your coverage number was worth.
I lead AI transformation for a global SaaS platform, and the verification lanes described here are what my agents actually merge through. If you’ve moved off coverage as a gate, or you’re running mutation testing at scale, I want to hear what it cost and what it caught. Find me on X, LinkedIn, or Telegram.