Read AI Delivery Metrics in Pairs: A DORA Stability Worksheet
A measurement worksheet that pairs delivery throughput with DORA stability signals and AI-contribution context, so you can baseline, compare, and investigate tradeoffs with your own data.
It is the Tuesday after the agent rollout review. PR throughput is up across the org, the slide says "developer velocity improving," and two on-call engineers in the room are quietly texting each other about the third rollback in the payments service this month.
Both things are true. That is the problem. The only reliable way to read AI delivery metrics is in pairs: every throughput measure read next to the stability measure that can absorb its gains. Change lead time next to change failure rate. Deployment frequency next to failed deployment recovery time. Read alone, a rising throughput number tells you nothing about whether the work was good, and DORA's own framing treats throughput and stability as a set of capabilities and outcomes rather than isolated scores you can win individually [1].
What follows is a worksheet, not a benchmark. You fill it in with your own repository, deployment, and incident data. There are no industry medians here to feel bad about, because comparing your payments service against an anonymized global percentile has never once told a team what to fix.
The Dashboard That Looked Like a Win
The throughput climb after an agent rollout is usually real. More pull requests get opened, and many of them merge. If your only instrumented number is merged PRs per week, you will see a line go up and you will be tempted to conclude something.
A merged-PR count carries no information about rework, review queue depth, where failures concentrate, or how long recovery takes. Those questions need their own fields, which is why DORA's capabilities catalog treats throughput and stability outcomes as separate things to measure rather than one composite score [1]. A delivery metric is a measure of motion. A stability metric is a measure of whether the motion was useful.
The pairing discipline is simple to state and annoying to maintain: no delivery metric gets presented, quoted, or screenshotted without its paired stability metric on the same line. If someone brings you a throughput chart with no failure rate next to it, send it back.
What the Four DORA Measures Tell You About AI Changes
Each of the four measures answers a different question when AI contribution rises in a repository. Naming the question keeps you from reading the measure as a score.
- Change lead time answers: is the bottleneck generation or approval? When generation gets fast and this number does not improve, your constraint was always review, not typing.
- Deployment frequency answers: is the extra merged work actually reaching users, or piling up behind a release train?
- Change failure rate answers: what share of that faster work needed correction? This is the metric that absorbs a throughput gain you did not earn.
- Failed deployment recovery time answers: when a change breaks, can anyone still diagnose it? This is where context loss shows up, long before anyone calls it technical debt.
The pairing logic matters more than the individual definitions. Lead time pairs with change failure rate because together they tell you whether speed came from removing friction or from skipping checks. Deployment frequency pairs with recovery time because shipping more often is only a gain if a bad release is still cheap to unwind.
One measurement trap deserves naming. If you measure lead time from first commit, the number will shrink the moment an agent writes the first commit quickly, even if the pull request then sits unreviewed for days. Record review latency as a separate field. DORA's capabilities catalog treats code review speed and change approval design as distinct levers on delivery performance, which is exactly why they need their own numbers rather than being blended into a lead time average [1].
What these four measures cannot tell you: code maintainability, whether reviewers actually understood what they approved, and whether knowledge about a module is concentrated in one person. Those need separate signals, and leaving them out of the worksheet is how teams end up surprised.
The Measurement Worksheet You Fill In Yourself
One worksheet per repository group. Not one per organization. A payments service and an internal analytics dashboard will never move together, and averaging them produces a number that describes neither.
| Field | What You Record | Where the Data Lives | Why It Matters |
|---|---|---|---|
| Baseline window | Exact start and end dates, pre-rollout | Frozen snapshot, not a live query | Without fixed dates, every comparison is arguable |
| Scope | Repos, branches, change types included | Repository inventory and discovery data | Undeclared scope is a common source of false conclusions |
| Delivery pair | Change lead time plus deployment frequency | Platform PR/merge API, CI deployment records | Motion, measured two ways |
| Stability pair | Change failure rate plus recovery time | Incident tool, rollback and revert records | The gain check on every delivery number |
| AI-contribution share | Share of merged changes tagged AI-assisted | Commit trailers or PR labels | A segmentation dimension, never a success measure |
| Review latency | Ready-for-review to first substantive review | Platform review timestamps | Where fast generation quietly goes to wait |
| Blocking-finding rate | Blocking findings per merged change, split from informational | Review tooling output [2] | Distinguishes real gates from comment volume |
| Exception count | Merges that bypassed a required gate, with owner | Ruleset bypass and audit logs [4] | A rising count can explain a rising failure rate |
| Revisit date | When this worksheet gets re-read | Calendar, with a named owner | An unrevisited worksheet is a dashboard |
Pull the data from systems you already run. Pull request and merge events come from the platform API. Deployments come from CI job records, not from someone's memory of the release. Incidents come from your incident tool with a consistent definition of what counts. AI attribution comes from commit trailers or PR labels, which we will get to.
Getting the scope column right is harder than it looks, because many organizations do not have an accurate list of which repositories are production-facing, which stack they run, and who maintains them. Repository discovery that identifies purpose, language versions, and team patterns is what turns the scope column from a guess into a statement. If you are assembling this by hand, start from the engineering analytics approach to measuring AI code ROI and narrow it to one service group first.
Choosing a Baseline You Can Defend
Set the baseline before the rollout. If you set it after, you are reconstructing history from whatever your dashboard currently computes, and dashboards recompute.
Make the window long enough to cover normal seasonality. A short pre-rollout window that happens to sit in a quarter-end change freeze will make any subsequent period look like an improvement. I have watched a team compare a post-rollout period against a much shorter pre-rollout baseline, then attribute a December deployment dip to the agent. The dip was the holiday calendar. The conversation cost the team most of a sprint and nearly killed a rollout that was working fine.
Record what else changed in the window. Team size, a framework migration, a new CI runner fleet, an org restructure. These are your confounders, and the only time you can name them honestly is before you have a result you want.
Freeze the baseline as a stored snapshot rather than a saved dashboard query. Metric definitions drift, incident severity thresholds get adjusted, and a live query silently rewrites your own history. A CSV in the repository beats a dashboard link.
Attributing AI Contribution Without Guessing
Attribution reliability varies by method, and many teams pick the least reliable one by default. Climb the ladder as far as your tooling allows:
1. Self-declared PR labels. Cheap, inconsistent, useful as a floor.
2. Commit trailers. Durable, greppable, survives in git history.
3. Bot or app identity on the authoring account. Strong signal when agents push under their own identity.
4. Tool telemetry. Most precise, hardest to join to merge and incident data.
Keep the disclosure in platform metadata rather than a tracking spreadsheet. Spreadsheets die when the person maintaining them changes teams, and you cannot join a spreadsheet to months of merge history. Recording AI assistance in platform metadata also keeps the record next to the code review evidence that secure-development practices expect you to produce for human-readable code, whoever or whatever wrote it [3].
A trailer convention plus an ownership rule gives you both attribution and accountability:
feat(billing): add proration for mid-cycle plan changes
Implements the proration path described in ADR-0042.
AI-Assisted-By: claude-code
Human-Reviewer: @mnovak
Reviewed-Paths: services/billing/**# CODEOWNERS
services/billing/** @payments-owners
db/migrations/** @platform-leads @dba-oncall
.github/workflows/** @platform-leadsCODEOWNERS makes the accountable human explicit per path and can be set as a required reviewer [5]. Repository rulesets then express required pull requests, required reviews, required status checks, and bypass restrictions as auditable configuration rather than as a convention people remember [4]. That combination is what makes an AI-authored merge record trustworthy: the agent identity cannot satisfy the review requirement it triggered.
One caution. Do not use share-of-lines-written-by-AI as an outcome metric. It rewards verbosity and tells you nothing about quality. Use contribution share only as a segmentation dimension, so you can ask whether high-AI-share repositories behave differently from low-share ones.
Reading the Pairs: Patterns and What to Investigate
| Pattern | Delivery signal | Stability signal | Investigate first | Likely owner |
|---|---|---|---|---|
| Earned gain | Throughput up | Failure rate flat | Sample merged PRs for review depth; confirm the gain is real | Eng manager |
| Borrowed gain | Throughput up | Failure rate up | The failing PRs and their incident postmortems | Service owner |
| Review bottleneck | Throughput flat, lead time up | Failure rate flat | Review latency by reviewer and path; check CODEOWNERS coverage [5] | Platform lead |
| Context loss | Throughput up | Recovery time up | Contributor concentration in the slow-recovery service | Tech lead |
| Gate erosion | Throughput up | Exception count up | The bypass log and who approved each exception [4] | Security owner |
Treat every pattern as a hypothesis, not a verdict. Write the question in the worksheet, name the artifact you will read, then go read it. A handful of pull requests with their full review threads will usually teach you more than another quarter of dashboard trend lines, and the reading takes an afternoon.
The artifacts worth pulling are specific: the failing pull requests themselves, review comment categories split into blocking versus informational [2], incident postmortems for the affected service, and the exception log. If you cannot locate those four things quickly, that gap is your actual finding.
Policy scope belongs in the same loop. The govern and map functions of the AI Risk Management Framework are where you document which repositories agents may act in and who owns the decision to widen that scope [7], and that scope statement is the same boundary your worksheet uses.
Where Governance Data Closes the Loop
A metric movement becomes actionable only when you can trace it to a control. "Change failure rate rose" is a trend. "Change failure rate rose in the two repositories where migration paths have no code owner and where merges used a ruleset bypass" is a work item with an owner [4][5].
That traceability depends on three artifacts many teams do not currently retain. First, the blocking-versus-informational split on review findings, so you know whether a gate fired or a comment was ignored [2]. Second, the exception record, including who approved the bypass and why, which rulesets can produce as platform data rather than screenshots [4]. Third, the decision record explaining why a policy was set the way it was, so the next person does not reverse it by accident.
This is where Guardian merge gates and typed decision records in Genie organizational memory earn their place in the worksheet. Gate configuration becomes the evidence for the stability column, and the decision record becomes the explanation when a reviewer asks why the gate exists. Mapping each artifact to the secure-development practice it supports means one record can answer both your monthly metrics review and the next customer security questionnaire, with build provenance attestations covering the artifact side of the same question [3][6].
When recovery time degrades in a specific service, contributor concentration is usually the missing variable. A service where one person wrote most of the code recovers slowly when that person is on leave, regardless of how good your agents are. Tracking bus factor and knowledge concentration alongside the stability pair turns "recovery got slower" into a staffing and documentation decision.
Running the Review Cadence Without Turning It Into Theater
Pick a fixed cadence, monthly or per sprint, and give it a named owner who presents the pairs. Not a dashboard tour. The owner walks through delivery and stability side by side, states which pattern each repository group matches, and stops.
Require one written hypothesis opened and one closed per review. Opened: "we believe the failure rate rise in billing comes from migration changes merging without owner review." Closed: "we read the billing PRs from the window; most had owner review; the pattern was test environment drift, not review coverage." Without this rule, the meeting produces trend lines and nothing else.
Keep a definition-change log. Every time a metric definition moves (what counts as a deployment, what severity counts as a failure, which repos are in scope), record the date and the change. Future comparisons are only honest if you can see where the ruler changed.
Do not tie these measures to individual performance reviews. It produces metric gaming quickly: smaller PRs to inflate counts, incidents reclassified downward, scope quietly shifted out of the worksheet. Measure the system, manage the people.
The team from the opening slide eventually segmented by service. The rollbacks were concentrated in two repositories, both with migration paths that had no code owner entry. The agent was not the cause. It was the load test that found a gap in the merge gate that had been there for a year.
Frequently Asked Questions
Do I need all four DORA measures to start?
No. Start with one pair. Change lead time plus change failure rate is the most informative first pair for AI-assisted work, because it catches the borrowed-gain pattern. Add deployment frequency and recovery time once the first pair is clean.
How long should the baseline window be?
Long enough to cover one full cycle of your normal seasonality, including any change freeze. If your team has quarter-end freezes, a short window will mislead you. Record the exact dates either way.
What if we cannot tag AI contribution reliably?
Run the worksheet anyway, with the attribution field marked as unavailable. The delivery and stability pairs are useful on their own. Add attribution when you can do it in platform metadata, and resist the temptation to backfill from memory.
Is a rising AI-contribution share good or bad?
Neither. It is a segmentation dimension. The useful question is whether high-share repositories show different stability behavior than low-share ones, and if so, what control differs between them.
How does this relate to merge gate design?
Directly. A rising exception count or a failure rate concentrated in unowned paths is a gate design finding. The risk-weighted approach to merge gates is the usual next step after the worksheet points at a specific control.
What to Do Next
Pick one repository group today, ideally one with real production consequence and a known owner. Fill in the scope and baseline window fields from data you already have, and snapshot them as a file in the repository rather than a dashboard link. That part takes well under an hour, and it fixes the most common measurement failure I see.
This week, start recording one signal you probably do not have: review latency from ready-for-review to first substantive review, separated from total lead time. It is the field that exposes whether faster generation actually produced faster delivery or simply relocated the queue.
Then set the revisit date and name the owner. When the next agent rollout review happens, the slide should carry two numbers per line, not one, and the engineers texting each other in the back row should be able to point at the pair that matches what they are seeing on call.
References
[1] DORA, "Capabilities catalog." https://dora.dev/capabilities/
[2] GitHub Docs, "Using GitHub Copilot code review." https://docs.github.com/en/copilot/using-github-copilot/code-review/using-copilot-code-review
[3] NIST, "SP 800-218: Secure Software Development Framework (SSDF) Version 1.1." https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-218.pdf
[4] GitHub Docs, "About rulesets." https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets
[5] GitHub Docs, "About code owners." https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
[6] GitHub Docs, "Using artifact attestations to establish provenance for builds." https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations/using-artifact-attestations-to-establish-provenance-for-builds
[7] NIST, "AI Risk Management Framework." https://www.nist.gov/itl/ai-risk-management-framework