Read the Reviewer's Own File Table Instead of Claiming It Has None - #625
Merged
Conversation
The PARTIAL caveat said the reviewer names no file list, so which file went unread cannot be read from the API. Measured over 348 review bodies here and 121 on Blog, that is wrong: 91 bodies carry a `| File | Description |` table, and every table row in the corpus belongs to one. The table is reported and never read as coverage, because the measurement says it cannot be. It names the whole changed set on partial and fully covered rounds alike, including all seven partials on Blog, so a full table is identical under both outcomes and discriminates neither. One round of the seven is the exception, stating 16 of 17 and naming 16 while omitting GOVERNANCE.md, and that is the only evidence on record that the unread file is a real file rather than an artifact of counting. `status` therefore prints one sentence beside the counts and takes no verdict from it. It names an omitted file only where the table is short by exactly what the counts leave unread and names nothing outside the diff, one round here naming GOVENANCE.md, a path no diff carries. The comparison is head-scoped like the counts, so a table from before a push, describing the diff that push replaced, reports as no table rather than as a stale list of unreviewed files. Exit 42 is unchanged in every case, and the merge decision stays the maintainer's. Fixes #623 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the pr_review.py partial-coverage caveat and supporting docs/tests to reflect that Copilot reviews sometimes include a per-file summary table, and to report what that table says (without letting it change the coverage verdict).
Changes:
- Parse the review body’s per-file summary table and compare it to the PR’s own changed-file list, reporting the result alongside
COVERAGE_IS_PARTIAL. - Extend the GraphQL full query to fetch the PR’s changed-file paths (windowed) and add test coverage for the observed table shapes and edge cases.
- Amend
TODO.md,scripts/README.md, and.github/copilot-instructions.mdto remove the incorrect “no file list” claim and document the measured limitations of the table.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| TODO.md | Updates the backlog entry with issue #623 and records the corrected measurement/disposition about file tables. |
| scripts/test_pr_review.py | Adds fixtures and a dedicated test suite covering parsing and reporting behavior for the review’s file table. |
| scripts/README.md | Updates documentation for partial-coverage handling to reflect the measured behavior of per-file tables. |
| scripts/pr_review.py | Implements file-table parsing, fetches PR file paths, and reports table-vs-diff comparisons under partial coverage. |
| .github/copilot-instructions.md | Updates the runbook guidance to note that the file table usually cannot identify the unread file and should be treated as a lead only in a narrow shape. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
ptr727
added a commit
that referenced
this pull request
Aug 8, 2026
One commit, `a706ddb` from #625, promoting the answer to #623 to `main`. ## What is promoted `scripts/pr_review.py` reads the Copilot review's own `| File | Description |` table and reports it beside a partial-coverage verdict, and the docs drop a claim that was false. - **The removed claim.** "The reviewer names no file list in these rounds, so which file went unread cannot be read from the API" was wrong. Measured over 348 review bodies on the newest 120 pull requests here and 121 on Blog, 91 carry a table and every table row in either corpus belongs to one. - **The table decides nothing, and the measurement is why.** It names the whole changed set on partial and fully covered rounds alike, including all seven partials on Blog, so it is identical under both outcomes. One round here states 33 of 33 and names 32, another states 61 of 62 and names 50, and a third names `GOVENANCE.md`, a path no diff carries. - **The one arm that locates a file.** #479 states 16 of 17 and names 16, omitting `GOVERNANCE.md`, and that is the only evidence on record that the unread file is real rather than an artifact of counting. `status` names an omitted file only in that shape, where the table is short by exactly what the counts leave unread and names nothing outside the diff. - **Head-scoped like the counts.** Three of the four partials here carry their table on the round before a push, describing the diff that push replaced, so those report as no table rather than as a stale list of unreviewed files. Exit `42` is unchanged in every arm, and the merge decision stays the maintainer's. ## State #623 is already closed by hand, since the keyword sat on a pull request into `develop`. Review on #625 was one round, `coverage=full shapes=ok threads=0 unresolved=0 suppressed=0 checks=6/6`, and the body was read rather than the digest trusted. **Merge as a merge commit, never a squash**, so `develop` and `main` stay level. Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #623.
Disposition
Amends "Find out which file a partial round skips, and why re-requesting never clears it."in the "A Programmatic Reading of a Copilot Review" cluster. #623 was filed after this repository's lastTODO.mdedit and appeared nowhere in the file, so selection step 1 was false until this change.What was wrong
The
COVERAGE_IS_PARTIALcaveat,scripts/README.mdand the runbook all said the reviewer names no file list in these rounds, so which file went unread cannot be read from the API. Measured over 348 Copilot review bodies on the newest 120 pull requests here and 121 on Blog, that is false. 91 bodies carry a| File | Description |table, and every table row in the corpus belongs to one of those tables.What the measurement says the table is worth
The issue proposed reading a table that names every changed file as evidence that the count is a reporting artifact. It cannot carry that, and the corpus is what says so.
GOVERNANCE.mdTODO.mdGOVENANCE.mdOn Blog every table names exactly the changed set, on all seven partial rounds and on every full round alike. A reading identical under both outcomes discriminates neither, so a full table is reported as corroborating nothing rather than as a miscount. #574 kills the converse, since a short table sits on a round that read everything.
What ships
Q_FULLselects the pull request's ownfiles, windowed at 100 withhasNextPagecarried, since a path outside a short window reads exactly like a path the reviewer left out and the record holds a pull request of 301 changed files.file_tablereads the table, quotations dropped for the reason the coverage line's are.head_tabletakes it from any round covering the current head and never from a round before a push. Three of the four partials here carry their table on the pre-push round, describing a diff that push replaced, and comparing that against the current changed files would name a file unreviewed on a stale list. Thirteen commits here carry more than one round, and on one of them a round with a table sits beside a round without, so which of the two the verdict reads must not decide whether a table is found.table_against_diffprints one sentence and decides nothing. It names an omitted file only where the table is short by exactly what the counts leave unread and names nothing outside the diff, that second guard existing because Clear the carried half of the prose backlog, and plan its re-vendor #606's typo would otherwise push the realGOVERNANCE.mdinto the omissions and report it as the file nobody reviewed.42is unchanged in every arm, and the caveat keeps its maintainer-decision framing, which is the issue's third suggestion.Verified against live data
Gates
228
test_pr_reviewcases, plustest_prose_lint,test_repo_gate,spec/audit.py --selftest,spec/validate.py, bothprose_lint.pyinvocations,markdownlint-cli2andeditorconfig-checker, all green locally.Generated with Claude Code