Engineering Quality Beyond Test Coverage: Metrics That Actually Matter

Test coverage percentage is a poor predictor of production reliability. Here are the indicators (Change Failure Rate, Review Depth Score, and rework rate) that actually tell you whether your codebase is healthy.

Connectory team|Updated 8 min

Picture a microservice that ships with 92% line coverage and, weeks later, causes a checkout outage. The race condition that brings it down was never tested: every line was covered, but the critical timing assumption between two async operations had zero assertions.

Coverage is a proxy. It tells you which lines a test runner touched, not whether your system behaves correctly under real conditions. Yet it remains the dominant metric in most engineering orgs because it's easy to measure, easy to enforce in CI, and easy to show on a dashboard.

Goodhart's Law applies directly here: when a measure becomes a target, it ceases to be a good measure. Developers learn to write tests that satisfy coverage thresholds without actually verifying behavior. Here's an example:

python
# 100% line coverage, zero meaningful assertions
def apply_discount(price, customer_tier):
    if customer_tier == "premium":
        return price * 0.85
    elif customer_tier == "standard":
        return price * 0.92
    return price

def test_apply_discount():
    result = apply_discount(100, "premium")
    assert result is not None  # passes, catches nothing
    result2 = apply_discount(100, "standard")
    assert result2 is not None  # same useless pattern

This test suite reports 100% coverage and catches nothing about whether the discount math is correct, whether negative prices are handled, or whether an unknown tier silently returns full price.

What Production Actually Tells You

The metrics that track production reliability come from DORA's research program, not from test runners. They measure outcomes, not proxies.

Coverage vs Leading Indicator Metrics
Coverage vs Leading Indicator Metrics

Change Failure Rate (CFR)

DORA defines change failure rate as "the ratio of deployments that require immediate intervention following a deployment," likely a rollback or a hotfix [1]. Many teams operationalize it as deployments that cause a rollback, hotfix, or service degradation within 48 hours. It's the most direct measurement of "did this change break production."

Track it over time and compare services against each other. A service whose changes regularly need intervention is telling you something coverage cannot.

CFR can be instrumented with two data sources: your deployment system (GitHub Actions, Spinnaker, Argo CD) records when a deployment happens, and your incident system (PagerDuty, OpsGenie) records when something goes wrong. Match deployment timestamps against incident creation timestamps. Rollbacks within 48 hours of a specific commit are your failures.

Most teams undercount CFR because they only track explicit rollbacks. Silent hotfixes (a quick config change, a feature flag flip, a database update that "fixed" a broken deployment) should count too.

Mean Time to Recovery (MTTR)

MTTR measures how quickly your team restores service after a failure. DORA's current version of this metric is failed deployment recovery time, "the time it takes to recover from a deployment that fails and requires immediate intervention" [1].

MTTR matters for quality measurement because it's a leading indicator of system complexity and operational readiness. Teams with high MTTR have deployed systems they don't fully understand. They lack runbooks, automated rollback, or clear ownership. These same teams often have high defect escape rates, because the system complexity that makes recovery slow is the same complexity that lets bugs slip through.

Track MTTR from incident created to incident resolved, segmented by service owner. Services with consistently high MTTR relative to peers are candidates for complexity reduction, not just better testing.

Defect Escape Rate

Defect escape rate is the ratio of bugs caught in production versus pre-production. As an illustration, a team catching 95% of defects before production has fundamentally different quality practices than a team where 40% of bugs reach users.

The challenge is attribution: you need both a pre-production bug tracker (Jira, Linear tickets marked as bugs found in review or QA) and a production incident system, then classify which environment surfaced each defect. This requires discipline in how your team logs bugs, but the signal it generates is worth the investment.

Teams that track defect escape rate tend to treat it as a system metric. When escape rate climbs, they investigate the category of defects getting through. Are they always in the same service? Same type of change? The pattern reveals where your quality process breaks down, which coverage percentage never can.

Review Depth Score

Review depth is the substantive quality of code review, not just whether a review happened. It's measured through comments per changed file, the time reviewers spent before approving, and whether review comments led to code changes before merge.

A pull request approved with zero comments in 45 seconds on 400 changed lines is not a review. It's a rubber stamp. Your CI gate passed, your branch protection triggered, and you have no real signal about whether a human understood what changed.

Review depth matters because it can act as a leading indicator. Check it against your own data: if low-depth periods are followed by higher CFR and defect escape rate, you have a signal worth watching. High depth means reviewers who ask questions, request changes, and understand the system.

GitHub's API exposes review comment count, approval timing, and whether changes were requested. The calculation is straightforward; the challenge is operationalizing it into a metric teams see regularly rather than an ad-hoc audit.

Git History as a Quality Signal

Your version control history contains quality signals that test coverage completely ignores.

PR Size Distribution. Review quality tends to degrade as PRs grow into the hundreds of changed lines; SmartBear advises reviewing "no more than 200 to 400 lines of code (LOC) at a time" because "beyond 400 LOC, the ability to find defects diminishes" [2]. Reviewers lose context, miss cross-file implications, and approve to unblock velocity rather than because they understood the change. Compare defect rates by PR size in your own history to find where the drop-off happens for your team.

Track the distribution of PR sizes over time, not just the mean. The mean is stable even when you have occasional 2,000-line PRs destroying review quality. The 90th percentile tells you whether your worst PRs are getting better or worse.

Bus Factor by Module. Measure what percentage of commits to each module in the last 90 days come from a single author. When one person owns most of the commits to a critical service, you have a single point of failure for both production incidents (they're on call) and knowledge (they quit). Coverage can be high; the system can still be brittle because only one person knows how it actually works.

Rework Rate. Rework rate measures what percentage of lines modified in a given week were also modified in the prior two weeks. A high or rising rate against your own baseline signals that your team is shipping code before it's ready: the first version lands in production, and the rework is the actual implementation.

Production Quality Feedback Loop
Production Quality Feedback Loop

High rework rate often follows external deadline pressure. The code shipped, it didn't break immediately (coverage was high), but it wasn't right, and engineers are quietly patching it in the background. That invisible technical debt never shows up in test coverage reports.

200-400
Lines of code SmartBear recommends reviewing at a time [2]
60-90 min
Review session length SmartBear pairs with that range [2]

90-Day Implementation Roadmap

Quality metrics are only valuable if they're instrumented automatically and visible consistently. Here's a phased rollout that doesn't require significant tooling investment upfront.

Weeks 1 to 2: Instrument CFR and MTTR. Connect your deployment pipeline to your incident system. Every deployment creates a record; incidents that open within 48 hours of a deployment create a linkage. Automate this as a daily job that writes to a shared dashboard. Even a basic spreadsheet with formulas works at the start; the discipline of checking it weekly matters more than the sophistication of the tool.

Weeks 3 to 4: Track Review Depth. Pull GitHub API data weekly: review comment count per PR, time-to-approval, and whether any reviews requested changes. Calculate a composite score per team (not per individual). Share the distribution in your engineering all-hands.

Weeks 5 to 8: Calculate Rework Rate and PR Size Distribution. Git blame gives you line authorship timestamps. Compare this week's changes against a two-week lookback window. PR size distribution is directly available from GitHub's PR diff API.

Weeks 9 to 12: First Scorecard Retrospective. Bring the three months of data into your next quarterly engineering review. Identify the bottom two services by CFR, the bottom two by review depth, and the top two by rework rate. These are your targets. Set specific improvement goals: "reduce CFR below 8% for payments service by end of Q3" is actionable; "improve quality" is not.

A suggested metric weighting for a composite quality score: CFR (30%), defect escape rate (25%), review depth (20%), rework rate (15%), PR size distribution (10%). Adjust weights based on your organization's biggest pain points.

Metrics should inform, not police
Never create individual developer leaderboards from these metrics. Aggregate at the team or service level. Individual metric tracking predictably drives gaming behaviors: developers learn to write smaller PRs that game size distribution without improving quality, or close incidents before service is really restored to improve MTTR statistics. Pair every metric with an action lever the team actually controls.

The Organizational Shift

Moving from coverage-centric to outcome-centric quality metrics requires changing what leadership celebrates. If, for example, a release decreases test coverage from 87% to 83% but reduces CFR from 18% to 6%, that's a quality improvement. If your org doesn't know how to read that signal, the engineer who made it will feel like they did something wrong.

The teams that make this transition successfully start by adding the outcome metrics alongside coverage, not replacing it immediately. Run both for one quarter. In every quality discussion, point to both. Watch which numbers move with the problems engineers actually experience. Outcome metrics like CFR and rework rate are built to do that; coverage is not.

In practice, the levers are concrete: PR size limits, requiring review comments before merge on large services, and tracking CFR weekly. Coverage can look fine while the real problems are happening; outcome metrics are the ones that surface them.

Frequently asked questions

Why is test coverage a poor predictor of production reliability?

Because it measures which lines a test runner touched, not whether the system behaves correctly under real conditions. A service can ship with 92% line coverage and still fail on a race condition whose timing assumption between two async operations had zero assertions: every line covered, nothing verified. Goodhart's Law applies directly, in that once coverage becomes the target it stops being a good measure, and developers learn to write tests that satisfy the threshold without checking behavior. A test asserting only that a result is not None reports full coverage on a discount function while catching nothing about whether the math is right, whether negative prices are handled, or whether an unknown tier silently returns full price.

Which metrics should replace coverage as the primary quality signal?

Outcome metrics rather than proxies. Change failure rate, which DORA defines as "the ratio of deployments that require immediate intervention following a deployment," likely a rollback or hotfix [1], is the most direct read on whether a change broke production. Mean time to recovery, which DORA now frames as failed deployment recovery time [1], acts as a leading indicator of system complexity and operational readiness. Defect escape rate is the ratio of bugs caught in production versus pre-production. Review depth score captures the substantive quality of review rather than whether one happened. Add them alongside coverage rather than removing coverage immediately, run both for a quarter, and watch which numbers move with the problems engineers actually experience.

How do you instrument change failure rate?

From two systems you already run. Your deployment tooling records when a deployment happens, and your incident system records when something goes wrong, so matching deployment timestamps against incident creation timestamps gives you the linkage. Rollbacks within 48 hours of a specific commit count as failures. The common error is undercounting by tracking only explicit rollbacks: silent hotfixes, a quick config change, a feature flag flip, or a database update that quietly fixed a broken deployment should all count.

What is a review depth score and how do you calculate it?

A composite of comments per changed file, the time a reviewer spent before approving, and whether review comments led to code changes before merge. A pull request approved with zero comments in 45 seconds across 400 changed lines is a rubber stamp, not a review: CI passed, branch protection triggered, and you have no signal that a human understood the change. GitHub's API exposes comment count, approval timing, and whether changes were requested, so the calculation is straightforward. The hard part is operationalizing it into something teams see regularly rather than an ad-hoc audit. Check it against your own history: if low-depth periods are followed by higher change failure rate and defect escape rate, you have a leading indicator worth watching. Reviewer trust calibration covers routing review depth by risk.

What quality signals live in git history?

Three that coverage ignores entirely. PR size distribution, because review quality degrades as changes grow; SmartBear advises reviewing "no more than 200 to 400 lines of code (LOC) at a time" since "beyond 400 LOC, the ability to find defects diminishes" [2]. Track the 90th percentile rather than the mean, because the mean stays stable while occasional 2,000-line PRs destroy review quality. Bus factor by module, measured as the share of commits to each module in the last 90 days from a single author, which is a single point of failure for both on-call coverage and knowledge. And rework rate, the share of lines modified in a week that were also modified in the prior two weeks.

What does a rising rework rate tell you?

That code is shipping before it is ready, with the first version landing in production and the rework being the actual implementation. A high or rising rate against your own baseline usually follows external deadline pressure: the code shipped, it did not break immediately because coverage was high, but it was not right, and engineers are quietly patching it in the background. That invisible debt never appears in a coverage report.

Should you track these metrics per developer?

No. Aggregate at the team or service level, because individual tracking predictably drives gaming: developers learn to write smaller PRs that game size distribution without improving quality, or close incidents before service is really restored to improve MTTR. Pair every metric with an action lever the team actually controls. The same caution applies to the composite score, where a workable starting weighting is change failure rate 30%, defect escape rate 25%, review depth 20%, rework rate 15%, and PR size distribution 10%, adjusted to your biggest pain points.

How long does it take to stand this up?

About a quarter, phased so no large tooling purchase is needed up front. Weeks 1 to 2 connect the deployment pipeline to the incident system and automate a daily job that writes change failure rate and MTTR to a shared dashboard; a spreadsheet with formulas is enough at first, because checking it weekly matters more than the tool. Weeks 3 to 4 pull review depth data from the GitHub API and share the distribution as a team-level composite. Weeks 5 to 8 calculate rework rate from line authorship timestamps and PR size distribution from the diff API. Weeks 9 to 12 run the first scorecard retrospective, picking the bottom two services by change failure rate and review depth and the top two by rework rate as targets, with specific goals rather than "improve quality."

References

[1] DORA, "DORA's software delivery metrics: the four keys." https://dora.dev/guides/dora-metrics-four-keys/

[2] SmartBear, "Best Practices for Peer Code Review." https://smartbear.com/learn/code-review/best-practices-for-peer-code-review/