Independent Checks for AI-Authored and AI-Reviewed Code
A review-signal matrix that separates authoring, automated review, deterministic checks, ownership review, and merge responsibility so you can see where independent evidence is still missing.
It is a familiar sequence. An agent opens the pull request. An automated reviewer leaves three comments, the author's agent fixes them, the conversation resolves. A human skims a green checklist and clicks merge. Four signals appeared on that pull request, and not one of them could have failed for a reason the authoring context did not already know about.
The fix is not more review. It is separating review into signals that can fail independently: authoring context, automated review commentary, deterministic checks, ownership review, and merge responsibility. A check adds information only when it can fail for reasons the author could not anticipate. Automated review comments are advisory feedback. Required status checks, approvals, and code-owner rules are the enforcement surface [5]. Keeping those two layers distinct is what makes the green checklist mean something.
This matters more now because the same files steer both ends. Claude Code loads project-level and user-level memory files as session context [1], AGENTS.md is a tool-neutral convention for repository-root agent guidance [2], and GitHub repository custom instructions influence Copilot responses in that repository, including code review [3][4]. When one convention file shapes generation and commentary, the reviewer inherits the author's blind spots. What follows is a five-layer matrix plus a checklist for finding where human judgment is still load-bearing. The goal is not slower merges. It is knowing which signals carry weight.
When the Author and the Reviewer Share a Model
Signal independence has a concrete test: could this check have failed for a reason the author's context did not contain? A test suite can. It runs against behavior, not against intent. A style reviewer reading the same AGENTS.md that produced the code cannot, at least not on the rules written in that file.
I have watched this collapse in practice. A team wrote an instruction rule requiring all new HTTP handlers to validate input through a shared schema module. The agent followed it. The automated reviewer, reading the same instructions, confirmed it. Nobody noticed that the shared schema module had stopped being the right answer for streaming endpoints two quarters earlier, because the rule was the only thing either side was comparing against.
That is not an AI failure. It is a correlated-signal failure, and it existed before agents. Agents just raise the volume until the correlation becomes the dominant pattern in your merge history.
The practical response is to classify every signal on a pull request as independent, correlated, or absent, then make sure at least one independent signal gates every merge.
The Five Review Signals and What Each One Can Prove
Each layer answers a different question. Confusing them is how teams end up with five signals and one real check.
| Signal | What it proves | What it cannot prove | Independent of author? | Enforcement |
|---|---|---|---|---|
| Authoring context | The change followed stated conventions | That the conventions are current or correct | No, it is the author's input | Reviewed as code |
| Automated review | Readable, convention-consistent code | Correctness, or anything its instructions omit | Correlated, configurable per repo [3][4] | Advisory comments only |
| Deterministic checks | Build, tests, lint, secrets, dependency facts | Design quality or business intent | Yes, fails on facts | Required status checks [5] |
| Ownership review | A human accountable for the subsystem looked | That the owner had time to look carefully | Yes, if the owner is not the author | Required code-owner review on sensitive paths [10] |
| Merge responsibility | A named identity accepted the change | Technical correctness | Yes, identity is recorded | Protected branch, no direct pushes [11] |
Deterministic checks are the most independent layer because they fail on facts rather than opinions. A failing test, a leaked credential, a dependency with a known advisory: none of these care what the instruction file says. DORA's capability set treats continuous testing and deployment automation as technical practices that improve delivery performance precisely because they are mechanical [12].
Automated review sits in the middle. It is genuinely useful for catching convention violations that linters cannot express, and GitHub documents it as a reviewer you can request manually or assign automatically [4]. But because custom instructions shape its output [3], its independence is a function of how the instructions were written and how recently they were checked.
Merge responsibility is the thinnest signal and the most often conflated with the others. It proves a named identity accepted the change. That is not nothing, and it is not a technical review.
Why Automated Review Should Never Be an Approval
Keep the feedback surface and the enforcement surface in different places. Comments are feedback. Approvals and required status checks are the gate [5]. The moment an automated reviewer's approval can satisfy a required-approvals rule, you have given the authoring context a vote on itself.
Then delete overlap deliberately. If a formatting or type rule already runs as a required status check, instruct the reviewer not to comment on it. Duplicated signals teach humans to skim, and a skimmed review is an absent review wearing a passing badge. Our notes on tuning custom review rules walk through phrasing conventions as checkable statements rather than style preferences.
A reasonable starting configuration for a default branch:
Require a pull request before merging
Require 1 approving review from a human identity
Require conversation resolution before merging
Require status checks: build, unit-tests, lint, secret-scan, dependency-review
Block force pushes and direct pushes, including automation accountsTrack which automated comments get resolved versus dismissed. A rule whose comments are consistently dismissed is not a review signal, it is noise with a cost. Rewrite the instruction behind it or delete the rule. That measurement is the only honest way to tell whether your review configuration is improving.
Stale Instruction Files Are a Review Signal Failure
Drift in AGENTS.md or a memory file does not announce itself. It steers generated code and review commentary in the same wrong direction, which is the worst possible failure shape because both sides agree.
Treat instruction files as code. Require review on changes to them. Add one line to the pull-request template: did a decision change, and did you update the instruction files in this pull request? That question catches more drift than a quarterly audit, because it fires at the moment the decision changes.
Write rules as short imperatives with a one-line rationale and a link out, not prose narrative:
## Data access
- Use the repository layer in `internal/store`. Do not call the database
driver directly. Rationale: migration safety, see ADR-0042.
- Never edit files under `gen/` or `vendor/`. They are generated.Then sample. Once a quarter, read a handful of recent agent-authored diffs looking for rules that were ignored or that now contradict reality, and tighten or delete them.
The structural fix is to stop keeping intent only in a file the agent also writes against. Typed organizational memory holds policies, decisions, ownership, constraints, and open questions server-side, so a reviewer can compare a change against recorded intent rather than against a repository file that may have drifted. That is what the institutional memory layer is for, and it is why review against intent and review against convention are different checks.
Tiering Ownership Review Without Pooling It on Two People
The symptom is recognizable. Agent pull requests outnumber human ones, required review funnels to the two engineers who understand the most subsystems, and queue depth becomes the real deployment gate.
Fix ownership mapping first. CODEOWNERS maps path patterns to users or teams and can automatically request review from owners [10]. Assign teams, not individuals, so load spreads and a departure does not orphan a directory. Then verify that every high-risk directory actually resolves to an active owning team.
Then tier the paths instead of applying one uniform gate:
- Sensitive tier: authentication, authorization, payments, data migrations, infrastructure as code, and CI workflow configuration. Required signals: deterministic checks, one human approval, and required code-owner review. Exceptions owned by the subsystem's engineering manager and recorded with a reason.
- Routine tier: application features, internal tooling, documentation, test fixtures. Required signals: deterministic checks plus one human approval. No owner gate, because owner gates on routine paths are what create the bottleneck.
- Generated or vendored paths: excluded from agent editing entirely by instruction file rule, and protected by a check that fails if those paths change outside a regeneration workflow.
- CI configuration specifically: treat as sensitive even when the diff looks trivial, because a workflow change can disable the only independent signal on every other pull request.
Rulesets support this layering, and when several apply to one branch the most restrictive result takes effect [5]. Branch protection remains documented alongside rulesets for teams already using it [11]. We cover tier design in more depth in risk-weighted merge gates by blast radius.
Watch review queue depth per owning team. When one team becomes the default reviewer for everything, your tiering has quietly collapsed back into a bottleneck.
Mapping the Matrix to Public Secure-Development Guidance
When a customer questionnaire asks how you develop software securely with AI assistance, answer by naming public practice sets rather than describing bespoke process.
Use NIST SP 800-218, the Secure Software Development Framework, as the control spine and map your pipeline practices to its practice identifiers [6]. Layer SP 800-218A on top, the SSDF community profile addressing generative AI and dual-use foundation models, which extends those practices rather than replacing them [7]. For the agent-specific threat surface, use the OWASP Top 10 for large language model applications, which covers prompt injection and insecure output handling [8]. That taxonomy is the right one when an agent reads repository files, issue text, dependency metadata, or fetched pages and then writes code: treat all of it as untrusted input.
Add controls that do not depend on the model behaving well. Scope agent tokens per repository at least privilege. Require agent changes to arrive as pull requests, never as direct pushes to protected branches [11]. Run secret scanning, dependency review, and static analysis on every pull request regardless of author. CISA's Secure by Design material frames the expectation that these properties are built in and owned by leadership rather than bolted on [9].
Record which practices are implemented, partially implemented, or consciously deferred. A matrix with honest partials survives scrutiny. A matrix claiming full coverage you cannot demonstrate does not.
Evidence That Survives an Auditor's Question
Pick the evidence unit first, usually a release tag or a deployment. Then list exactly what must be shown for it: who or what authored each change, who approved it, which checks ran and passed, and that the shipped artifact came from the reviewed source.
Artifact attestations close the last link. A GitHub Actions workflow can generate provenance linking a built artifact back to the workflow and source that produced it, with a documented verification command so the link is checkable later rather than asserted [13]. Document that command in your runbook next to the attestation step, because an unverifiable attestation is just metadata.
# Verify a published artifact's provenance before trusting the release
gh attestation verify ./dist/app-1.4.2.tar.gz --repo your-org/your-repoGive automation its own account or app identity so commit authorship, pull-request authorship, and human approval are separable in the record [13]. Without that, agent-authored and human-approved collapse into one name and the independence you designed becomes unprovable. Our write-up on the AI code authorship provenance chain covers the attribution detail.
Export the control configuration itself into version control: rulesets, required checks, and CODEOWNERS. That makes the state of the gate at merge time reconstructable months later. Then pick a target SLSA build level, assess the gap honestly, and record the remaining work rather than claiming a level you cannot verify [14].
A Checklist for Finding Your Collapsed Signals
This takes about an hour and it is the single most useful audit you can run this quarter.
1. Pull ten recently merged pull requests from your busiest repository, mixing agent-authored and human-authored changes.
2. Label each signal on each one as independent, correlated with the authoring context, or absent. Use the five rows from the matrix table.
3. Flag every merge where all passing signals trace back to the authoring context. These are your collapsed merges.
4. Confirm ownership resolution for each high-risk directory: does it map to an active team, or to someone who left?
5. Verify agent identity is distinct in the commit and pull-request record for at least one agent-authored change.
6. Check one release end to end: authorship, approvals, checks that ran, and artifact-to-source linkage.
If step three produces more than one or two flags out of ten, your deterministic check layer is thinner than your merge history suggests.
Frequently Asked Questions
Can an AI reviewer count as the required approval?
Keep it advisory. Rulesets let you require approving reviews separately from status checks [5], so configure human approval as the approval requirement and let automated review post comments. The automated reviewer's instructions come from the same repository that shaped the code [3].
Does requiring code-owner review on everything solve independence?
No, it trades one problem for another. Universal owner gates pool review on the few people who own the most paths. Tier it to sensitive directories and let routine paths move on checks plus one approval [10].
What if our tests are weak? Are deterministic checks still the independent layer?
They are still the most independent layer available, but a thin test suite means thin independence. Strengthening tests on sensitive paths buys more real review than adding another reviewer.
Which framework should we cite in a security questionnaire?
SP 800-218 as the spine, SP 800-218A for the generative-AI additions, and the OWASP LLM Top 10 as the agent threat list [6][7][8].
What to Do This Week
Run the ten-pull-request audit described above and count how many merges had zero independent signals. That number is your baseline.
Then start tracking one metric monthly: share of merges with no independent signal. Not review coverage, not comment volume, not approval counts. The share of changes that shipped where every passing check traced back to the context that wrote the code.
The sequence in the opening paragraph will keep happening, and that is fine. Agents writing pull requests and automated reviewers commenting on them is a reasonable way to work. What makes it safe is knowing that somewhere in that green checklist, at least one check could have said no for a reason nobody involved had thought of.
References
[1] Anthropic, "Claude Code: Manage Claude's memory," 2025. https://docs.claude.com/en/docs/claude-code/memory
[2] AGENTS.md, "Open format for guiding coding agents," 2025. https://agents.md/
[3] GitHub Docs, "Adding repository custom instructions for GitHub Copilot," 2025. https://docs.github.com/en/copilot/how-tos/configure-custom-instructions/add-repository-instructions
[4] GitHub Docs, "Using Copilot code review," 2025. https://docs.github.com/en/copilot/using-github-copilot/code-review/using-copilot-code-review
[5] GitHub Docs, "About rulesets," 2025. https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/about-rulesets
[6] NIST, "SP 800-218: Secure Software Development Framework (SSDF) Version 1.1," 2022. https://csrc.nist.gov/pubs/sp/800/218/final
[7] NIST, "SP 800-218A: Secure Software Development Practices for Generative AI and Dual-Use Foundation Models," 2024. https://csrc.nist.gov/pubs/sp/800/218/a/final
[8] OWASP, "Top 10 for Large Language Model Applications," 2025. https://owasp.org/www-project-top-10-for-large-language-model-applications/
[9] CISA, "Secure by Design," 2025. https://www.cisa.gov/securebydesign
[10] GitHub Docs, "About code owners," 2025. https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
[11] GitHub Docs, "About protected branches," 2025. https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches
[12] DORA, "Capabilities: continuous delivery and technical practices," 2025. https://dora.dev/capabilities/
[13] GitHub Docs, "Using artifact attestations to establish provenance for builds," 2025. https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations/using-artifact-attestations-to-establish-provenance-for-builds
[14] SLSA, "Security levels (v1.0 specification)," 2023. https://slsa.dev/spec/v1.0/levels