Reviewer Trust Calibration: Scoring AI-Authored Pull Requests
A seven-factor rubric for scoring author type, blast radius, test coverage, dependency risk, and policy conflict so each pull request routes to lightweight, standard, or deeper human review.
It is 4:47pm on a Thursday. A pull request lands with a clean title, twelve changed lines, and a green check next to every status. The reviewer skims it, sees no obvious problem, and approves in under a minute. Two weeks later, an incident review discovers that those twelve lines changed how a refresh token is cached, and the agent that wrote them had no way of knowing the team had already rejected that approach.
Nobody was negligent. The reviewer applied the only heuristic available in the moment: the diff was small, the checks were green, and the queue was long. Review depth was decided by diff size and reviewer availability, not by risk.
The fix is to score each pull request before you decide how much human attention it needs. Rate it on author type, blast radius, test coverage change, dependency delta, repository context fit, policy conflict, and reversibility, then route it to lightweight, standard, or deeper review. "Was this written by AI?" is a weak first question. "What breaks if this is wrong, and how hard is it to undo?" is the question that predicts real damage. This article gives you a seven-factor rubric you can adapt, wire into GitHub rulesets, and defend to an auditor.
The Approval That Nobody Actually Read
Most teams already run some form of trust calibration. It is just undocumented and inconsistent. One reviewer reads agent-authored diffs line by line and waves through colleagues' work. Another does the reverse, assuming the tooling caught anything mechanical. Neither behavior is written down, so neither can be improved.
Author type does carry information. An unattended agent working from an issue description has no memory of the conversation where your team ruled out client-side token caching, unless that decision lives somewhere the agent reads. But author type is a single input, not a verdict. An agent-authored typo fix in a README and an agent-authored change to your authorization middleware deserve wildly different treatment, and a rushed human change to a payment path can be more dangerous than either.
What makes scoring worth the effort is that it converts a vague feeling into a routing decision you can enforce. GitHub rulesets can require pull requests, a set number of approvals, required status checks, and code owner review before merge, and multiple rulesets can layer on the same branch [1]. CODEOWNERS assigns review requests by path pattern and can be made a requirement through those rules [2]. Once you know a pull request scores high, the enforcement surface already exists. The missing piece is the score.
Seven Factors That Actually Predict Review Depth
Keep the factor list short enough that a reviewer can hold it in their head. These seven cover the failure modes I have seen actually reach production.
- Author type. Human, human with an assistant, agent under supervision, or unattended agent. Do not infer this from code style. Record the agent's role in commit and pull-request metadata so reviewers can distinguish assisted work from unattended automation.
- Blast radius. Authentication and authorization, data migrations, money movement, CI workflow files, and build configuration outrank ordinary feature code. Require heavier review where blast radius justifies it rather than everywhere [2].
- Test coverage change. New behavior with no new assertions is a signal regardless of who wrote it. A diff that adds a branch and adds no test is telling you where the reviewer should start reading.
- Dependency and supply-chain delta. A new package, a major version bump, or a shifted transitive tree changes the code you are responsible for far beyond the diff shown in the pull request.
- Repository context fit. Does the change match the conventions the repository actually publishes? Agent context now lives in reviewed files: an AGENTS.md at the repository root for build, test, and style expectations [3], project-level CLAUDE.md memory that Claude Code loads automatically [4], and repository custom instructions applied to Copilot requests in that repository [5]. If a change contradicts those files, that is a scoring event.
- Policy conflict. Does the change contradict a recorded decision, revive a rejected alternative, or violate a stated constraint? This is the factor most teams score by memory, which means they score it inconsistently.
- Reversibility. A feature flag you can flip in thirty seconds is not a schema change you cannot roll back. Reversibility should lower the score, and irreversibility should raise it.
Notice that only one of the seven is about the author. That ratio is deliberate. Calibration that keys mainly on author type produces a reviewing culture that suspects agents and trusts humans, which is precisely backwards for a hurried Friday change to a billing path.
Turning Factors Into a Score You Can Defend
Use a small additive scale. Zero to five per factor, summed, with published band thresholds. Resist weighted formulas with coefficients nobody can explain in a standup, because a score a reviewer cannot reproduce is a score they will ignore.
| Factor | Low (0-1) | Elevated (2-3) | High (4-5) | Why it matters |
|---|---|---|---|---|
| Author type | Human author, reviewed locally | Human with assistant, or supervised agent | Unattended agent, no human in the loop | Determines whether anyone has read the change with intent before you |
| Blast radius | Docs, tests, internal tooling | Feature code with contained callers | Auth, migrations, payments, CI workflow or build config | These paths convert a small mistake into an outage or a breach |
| Test coverage | Tests added or unchanged behavior | Behavior change with partial assertions | New branch or new behavior, no new tests | Absence of assertions tells you nobody proved the intent |
| Dependency delta | No manifest change | Patch or minor bump | New package, major bump, transitive shift | Expands your attack surface beyond the visible diff |
| Context fit | Matches AGENTS.md and repo conventions | Minor drift from conventions | Contradicts published in-repo constraints | Drift compounds silently across agent sessions |
| Policy conflict | No recorded decision touched | Adjacent to a recorded decision | Revives a rejected alternative | Re-litigated decisions are the most expensive kind of rework |
| Reversibility | Flag-gated or trivially revertable | Revert needs coordination | Irreversible or data-destructive | Recovery cost, not likelihood, sets the review budget |
Publish the thresholds in the repository next to your other context files. A workable starting split: 0 to 7 total goes lightweight, 8 to 15 goes standard, 16 and above goes deeper. Calibrate those numbers against your own history rather than adopting mine.
Then add one override rule, because totals hide danger. Any single high-signal factor on a protected path escalates the pull request regardless of total score. A change that scores 5 on blast radius and 5 on irreversibility does not get lightweight treatment because six other factors were clean.
The failure mode to guard against is scoring theatre: a reviewer who merges first, then fills in the rubric to justify the tier they already chose. If the score is computed after the merge decision, you have added paperwork and gained nothing. Score at pull-request open, from metadata and diff facts, before a human forms an opinion.
Three Review Tiers and What Each One Owes the Author
Tiers only work if each one is a real commitment. The author needs to know what they are getting and when.
| Tier | Required approvals | Required checks | Evidence produced | Best for |
|---|---|---|---|---|
| Lightweight | One approval | AI pass plus status checks | Approval record and check run | Docs, test-only changes, isolated internal tooling |
| Standard | Code owner on the affected path | Status checks plus resolved AI findings | Approval, check run, resolved-comment thread | Feature code with contained blast radius |
| Deeper | Two named reviewers including a domain owner | All standard checks plus threat review | Written decision record and provenance attestation | Auth, migrations, money movement, CI and build config |
| Escalated exception | Deeper tier plus named exception owner | Same as deeper, plus documented bypass | Bypass grant in the audit log | Time-critical fixes where a gate must be waived |
For the deeper tier, add a threat review step when agents read untrusted text. If your agent tooling consumes issue bodies, pull-request comments, dependency README content, or fetched web pages, use the OWASP Top 10 for Large Language Model Applications as the checklist, especially prompt injection and insecure output handling [6]. Those are not hypothetical categories for a system that lets an issue description influence a code change.
Enforce the tiers with configuration, not convention. Move branch policy into organization-level rulesets so new repositories inherit pull-request, review, and status-check requirements by default [1]. Keep an explicit bypass list rather than informal exceptions, and review who holds bypass permission alongside your other access reviews. Protected branch settings and ruleset behavior are documented well enough that there is no reason to run this as tribal knowledge [7].
Where AI Review Belongs in the Score, and Where It Does Not
AI review is an input into the score. It is not an approval. GitHub documents Copilot code review as a reviewer you request on a pull request, and its feedback can be guided by repository custom instructions so comments reflect local conventions [8][5]. That is genuinely useful for convention drift and missing tests. It does not carry accountability, and no configured merge gate should treat it as though it does.
The factor that AI review can improve most is policy conflict, because that is the one humans score from memory. Scoring it reliably requires that decisions, rejected alternatives, ownership, and constraints exist as retrievable records rather than Slack archaeology. This is the gap Genie's typed institutional memory is built to close: chosen options, the alternatives you ruled out, and the reason, held as structured records instead of prose scattered across wikis. Once those records exist, a review pass can check a diff against them. SlopBuster's context-aware review reads repository context and recorded organizational intent, so a change that revives a rejected approach surfaces as a finding rather than as a reviewer's vague recollection. Guardian can surface review and policy signals for the team. Configure the selected merge tier in GitHub rulesets or branch protection, and record any exception through your team's documented approval process.
Whatever tooling you use, measure it. Track which AI findings reviewers accept and which they dismiss, and prune the instructions that generate noise so the remaining signal stays worth reading [8]. When most comments get dismissed, you are training your team to scroll past the review surface, which costs you the findings that mattered.
Worked Examples: Two Pull Requests, Two Very Different Tiers
Example one: retry logic on a payments webhook
An unattended agent opens a pull request adding retry logic to a payments webhook handler. It adds one HTTP client library, changes no tests, and reintroduces at-least-once delivery handling that the team explicitly replaced with an idempotency key check six months ago.
Score it: author type 5 (unattended), blast radius 5 (money movement), test coverage 4 (new branch, no assertions), dependency delta 4 (new package), context fit 3, policy conflict 5 (revives a rejected alternative), reversibility 3. Total 29, and three factors are high-signal on a protected path. This is a deeper-tier review: two reviewers including the payments domain owner, a threat review because the agent read the issue text that triggered the change, and a written decision record explaining why the idempotency decision is being revisited or upheld.
Example two: fixture generator update
The same agent updates a test fixture generator in a test-only directory, adds three assertions, and touches no manifest.
Score it: author type 5, blast radius 0, test coverage 0, dependency delta 0, context fit 0, policy conflict 0, reversibility 0. Total 5. Lightweight tier: AI pass, required status checks, one approval. Author type alone did not escalate it, and it should not have.
Give reviewers a paste-in artifact so scoring takes under a minute:
## Trust calibration
Author type: [0-5] agent role from PR metadata
Blast radius: [0-5] paths touched
Test coverage: [0-5] new behavior vs new assertions
Dependency delta: [0-5] manifest + lockfile change
Context fit: [0-5] AGENTS.md / repo conventions
Policy conflict: [0-5] recorded decisions touched
Reversibility: [0-5] revert cost
Total: __ Tier: lightweight / standard / deeper
Override triggered: yes / no Reason: ______Both examples leave behind the same evidence set, which is the point. The pull request and its approvals, the required checks that passed, the ruleset in force at merge time, and signed provenance for the artifact that shipped. GitHub Actions artifact attestations generate signed provenance linking an artifact to the workflow and source that produced it [11], and the SLSA provenance specification defines the format and fields those claims carry [12]. When an auditor asks who approved a specific change and how the release traces back to it, three verifiable records beat three recollections.
Frequently asked questions
Does trust calibration slow delivery?
Tiering exists so that heavier review lands only where blast radius justifies it, which should reduce the total review load compared with treating every pull request as risky. Verify it rather than assuming it. Watch throughput and change failure signals using DORA's published measures after you tighten controls, and remove gates that add queue time without catching defects [10].
How do we know an agent authored the change?
Record it. Commit and pull-request metadata should carry the agent's role so reviewers can tell supervised assistance from unattended automation. Inferring authorship from code style is guesswork that will be wrong in both directions.
Do we need a separate AI policy?
Usually not. Map your existing controls to NIST SP 800-218 practice groups first, then use SP 800-218A, the community profile that augments the SSDF for generative AI and dual-use foundation models, to add the practices where model-assisted development changes the evidence you need [13][14]. A standalone AI policy that never reconciles with your SDLC becomes shelfware. CISA's Secure by Design material is useful for framing ownership of secure outcomes at the leadership level [15].
What if reviewers disagree with the score?
Name the exception owner before the disagreement happens, and record the outcome where auditors can find it. Bypasses should be explicit ruleset grants that appear in the organization audit log, not a Slack thread [9].
How often should trust calibration thresholds change?
On a fixed cadence tied to observed misses. Quarterly works for most teams. Bring two inputs to the review: pull requests that merged at a lower tier than their score justified, and accepted-versus-dismissed rates on AI findings.
Start With One Repository and One Protected Path
You do not need an organization-wide rollout to get value from this. Do three things in the next thirty minutes.
1. Pick your highest-risk path in one repository. Auth, migrations, or the CI workflow directory.
2. Write the three tier thresholds into the repository as a short file next to your AGENTS.md, and link it from the pull-request template.
3. Confirm CODEOWNERS resolves to a staffed team for that path, not an individual who changed roles last year [2].
The signal to start tracking this week: pull requests that merged at a lower tier than their score justified. Count them. That number is your calibration gap, and it is more informative than any tool evaluation you could run.
In week two, move the branch policy into an organization-level ruleset so new repositories inherit the tiers by default [1], and keep the bypass list short and reviewed.
Go back to that 4:47pm approval. Under this rubric, the token refresh change scores high on blast radius, high on policy conflict, and high on missing tests. It never reaches a single tired reviewer with a green checkmark and a long queue. It routes to the auth owner with a note explaining which recorded decision it contradicts.
Review depth should be a recorded decision, not an accident of who happened to be online.
References
[1] GitHub Docs, "About rulesets," 2025. https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets
[2] GitHub Docs, "About code owners," 2025. https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
[3] AGENTS.md, "Open format for guiding coding agents," 2025. https://agents.md/
[4] Anthropic, "Claude Code: Manage Claude's memory," 2025. https://docs.claude.com/en/docs/claude-code/memory
[5] GitHub Docs, "Adding repository custom instructions for GitHub Copilot," 2025. https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions-for-github-copilot
[6] OWASP, "OWASP Top 10 for Large Language Model Applications," 2025. https://owasp.org/www-project-top-10-for-large-language-model-applications/
[7] GitHub Docs, "About protected branches," 2025. https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches
[8] GitHub Docs, "Using GitHub Copilot code review," 2025. https://docs.github.com/en/copilot/using-github-copilot/code-review/using-copilot-code-review
[9] GitHub Docs, "Reviewing the audit log for your organization," 2025. https://docs.github.com/en/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/reviewing-the-audit-log-for-your-organization
[10] DORA, "DORA metrics: the four keys," 2025. https://dora.dev/guides/dora-metrics-four-keys/
[11] GitHub Docs, "Using artifact attestations to establish provenance for builds," 2025. https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations/using-artifact-attestations-to-establish-provenance-for-builds
[12] SLSA, "Provenance specification v1.0," 2025. https://slsa.dev/spec/v1.0/provenance
[13] NIST, "SP 800-218: Secure Software Development Framework (SSDF) Version 1.1," 2022. https://csrc.nist.gov/pubs/sp/800/218/final
[14] NIST, "SP 800-218A: Secure Software Development Practices for Generative AI and Dual-Use Foundation Models," 2024. https://csrc.nist.gov/pubs/sp/800/218/a/final
[15] CISA, "Secure by Design," 2025. https://www.cisa.gov/resources-tools/resources/secure-by-design