fix(provenance): discharge the rubric-v3 re-score, and the answer key it found in the rubric - #3472
Conversation
…es listing as unenforced Two design gaps from the report-only sweep (#3465): Gap 1: the evidence-tier table had no row for a comparison basis that is an in-repo vendored snapshot (live fetch failed, only the committed copy under vendor/ with a declared upstream ref and sync date was available). New section in reference/source-fetch.md states the rule: the finding caps at source-fetched-similar, is never fix-eligible, records source.route: vendored-snapshot with the snapshot path, upstream ref, sync date and each failed fetch, and the follow-up is human. SKILL.md step 9 carries the cap so tier mapping cannot miss it. No existing tier row in reference/rubric.md was touched. Gap 2: the not-found searched-surfaces listing is required by prose in three places but scripts/emit-findings.sh has no field, count, or schema for it, and the relay boundary withholds not-found findings from the findings file entirely. Decision (b): record the limitation rather than assert a capability. reference/source-fetch.md now states the listing is prose-only and never validation evidence; SKILL.md's "does not" bullet says the same. Also flagged in reference/dispositions.md, where a reader of the fix contract sees it: a corpus file can be the rendered output of a generator whose source of record lives outside the markdown corpus (the sweep found one at docs/native-surfaces/records.json), so a disposition applied to the rendering edits a never-hand-edit file. Flag only; no guard built. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
"may" is a month name and an ordinary English modal verb. Both stamp
detectors matched it bare, so prose like "the first read may raise a
permission prompt" became a stamp candidate whose date form could not be
parsed and landed in the declined bucket. A reader adjudicating that
bucket could not tell such a line from a real stamp we failed to parse:
17 of the 22 month-name declines in the 2026-08-28 corpus sweep carried
the word, and 13 of them were the modal sitting inside a keyword window.
"may" now needs a digit beside it before it counts as a date. Every date
form carries one ("may 2026", "may 17", "17 may") and the modal does not.
The tightening is deliberately narrow: the other eleven months still
match bare, because over-reporting into a bucket a human reads is the
safe direction and under-reporting is not.
Both scripts carry the month list on purpose, so both change together and
the suites pin the agreement directly: extract-breadcrumbs.test.sh runs
check-stamps.sh over the shared fixture and asserts the two candidate
counts match. A real "May 2026" stamp is asserted as still detected in
both suites, which is the regression this fix could plausibly cause.
Corpus at 1,352 files, --as-of 2026-08-28: candidates 544 -> 527,
declined 45 -> 28, month-name declines 22 -> 5, bare-year declines 23
unchanged, parsed 499 unchanged, findings 0 unchanged. Every one of the
17 dropped lines carried the modal; the 5 that remain are genuine
month-name stamps. The extract-breadcrumbs inventory moved by the same
17 lines (534 -> 517), so the two definitions stayed in step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
Composed centrally rather than by the workers that did the work: three agents ran concurrently in one tree, and having each write its own entry is how a sibling's commit got corrupted earlier in this branch's history. Covers the vendored-snapshot tier rule (caps at source-fetched-similar, records source.route, never fix-eligible), the modal-"may" false positive (three sites, declines 45 to 28, parsed and findings both unchanged), the searched-surfaces listing recorded as prose-only and unenforced, and the generated-output warning in the fix contract. Minor rather than patch: the tier contract gains a case it could not previously express, which is additive guidance a consumer can rely on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
Version 3's version-history paragraph recorded the expected re-score tally, the panel size, and an enumeration of which golden case turns on which criterion, including the one case the scope change exists to restate. The pipeline inlines the whole rubric into every judge prompt at the judgment step, so all thirty judges in the version-3 re-score read the prediction before grading, and that run had to withdraw its claim of a blind panel. Found by that run's own fresh-context verifier, which returned FAIL on the method while confirming the arithmetic. I wrote that paragraph to keep the figures from sitting under a cloud they did not deserve. Putting it in the file judges read at judgment time is what made a blind measurement against the rubric impossible. The prediction and the enumeration are changelog material and now live there. The rubric keeps the criteria, the carve-outs, the scope rule, the worked examples and the tier table, and now states that a judge should be able to read all of it and still not know the answer. Also records the re-score result (8 tp / 0 fp / 0 fn / 2 tn, no verdict moved, no class fix-eligible because every class is below min_n_per_class), the sensitivity panel that re-judged c04 without the leaking preamble and returned the same verdict on the same reasoning, and a second leak of the same kind in the c08/c09/c10 shared source.md. That one is deliberately not fixed here: the line sits inside the text the fingerprint module compares, so removing it moves the containment and span figures those cases record. The fix and a re-score are one atomic change and splitting them would leave a measurement that no longer reproduces from its own fixtures. Refs #3465. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
…made stale Both found by the fresh-context verifier on the "may" fix, which passed the change itself. The changelog justified deferring the underscore-boundary fix by citing a `last_verified_2026_08` stamp that would be lost. No such token exists anywhere in the repo. The verifier checked, found nothing, and supplied the real line: plugins/work-items/skills/track/actions/add.md:104 carries "last_checked": "2026-04-08", which parses today and would indeed be lost. The conclusion held; the evidence for it was invented, and an invented exhibit in an entry about provenance is the wrong defect to ship. Also records what the verifier established and the deferral rationale had not: a token-boundary fix repairs exactly 1 of the 23 bare-year declines, because the rest carry years that are already token-boundaried. That is the real argument for deferring, and it is stronger than the one given. The window-slack comments in both scripts justified their 9 characters as "one short of the longest form matched below". That stopped being true when the "may" rule landed: unlike every other form it is unbounded in length, so a contrived "may" plus 10 or more non-letters plus a year, starting at exactly wlen, runs past the slack and stops being a candidate. Latent, not live: no such line exists in the corpus, parsed and findings both stayed flat, and the ordinary "may 2026" at the same offset is still caught. Widening the slack would move the RSTART <= wlen boundary two earlier commits tuned, for a form nobody writes. Both comments now say fixed-length, and the rule carries the edge and the reason it is left. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
|
Claude finished @kyle-sexton's task in 2m 56s —— View job Code review complete
Scope: reviewed the full diff ( Finding posted inlineOne high-signal correctness issue, posted as an inline comment on
What I checked and found sound
Everything else in the diff (rescoring narrative, design-gap docs) is prose/documentation with no executable surface to verify beyond the one inconsistency above. |
|
Claude finished @kyle-sexton's task in 1m 7s —— View job Security review complete ✅
No security-relevant surface in this diff. All 10 changed files are within
No CRITICAL / IMPORTANT / SUGGESTION findings to report. |
|
Last security-reviewed head: |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Claude has reviewed this PR 5 times. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 52307d7717
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…red number The review lane found the script comments saying 13 of 22 while the changelog said 17 of 22: same measurement, same corpus, same date, two numbers depending on which file you read. Measuring turned up a third. The shipped figure is 19 of 24, over 1,352 files at --as-of 2026-08-28 on this branch head. The 17 of 22 was measured correctly, but against the branch base before it was repaired: that base sat two commits behind main and so lacked merged changelog entries whose own prose contains the modal. Repairing the base moved the count to 19 of 24. The 13 matched no tree. All three sites now carry the measured number and say the count is tree-dependent, because a corpus that includes this repository moves when this repository gains prose. That is the same effect 0.3.2 records, showing up one layer further out: not a stale absolute this time but a figure that was accurate at the commit it was taken on and stopped being so when the base changed underneath it. Reworded to avoid an apostrophe: the comment sits inside the single-quoted awk program, and "repository's" terminated the string. Caught by shellcheck before commit rather than in CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
|
Claude finished @kyle-sexton's task in 5m 34s —— View job Code review complete
Scope: this is a re-review at the new head ( Finding posted inline (new)Digitless "May" stamps silently disappear instead of declining, posted on Before this PR, The other eleven months are unaffected and still over-report into the declined bucket as intended. Only digitless "may" moved the other direction — from over-reporting to silent non-reporting — which is the direction This is a corpus-realistic pattern ("Verified this May", "Checked last May") the design otherwise explicitly tries to keep visible, so it's worth a decision: either accept the narrower blind spot as intentional (and say so in the comment/changelog), or keep digitless "May" landing in the declined/inventoried set the way the other eleven months do, distinct from a modal "may" with no stamp keyword nearby at all (which is the case this fix is actually meant to suppress). What I checked and found sound
|
|
Claude finished @kyle-sexton's task in 1m 18s —— View job Security review complete ✅
Skip gate hit — no new security-relevant surface since the last review. Last security-reviewed head:
No executable logic, permissions, secrets, auth, injection surface, or GitHub Actions config changed in this delta. This is the fix for the inline figure-mismatch finding from the prior code-review pass, not a new logic change, so the prior security review (no CRITICAL/IMPORTANT/SUGGESTION findings) still applies in full. Nothing new to report. |
… paths Requiring a digit beside "may" kept the modal out of the declined bucket and took a real stamp form with it. "Verified this May" carries no digit, so keyword_window() returned "" and the caller dropped the line at `if (win == "") next`, before it could reach n_dec++ and be classified. is_stamp() returned 0 the same way. The line was a candidate declining as a month-name form, visible to a human reading the declined bucket, and it became invisible to both audit paths at once. That is under-reporting, the one direction this detector must not move in, and it was inconsistent: "Verified in June" still matched bare and still declined. Either signal now counts as a May date, and either alone is enough: a digit beside it, as before, or a capital M read from the ORIGINAL line rather than the lowered copy the rest of the scan works from. Edited prose capitalises the month and not the modal, and tolower() was throwing away the only signal that separated them. Both scripts get one may_form() and all three sites call it: keyword_window(), the classifier that sets the decline reason, and is_stamp(). Two costs, measured over the 1,352-file corpus at --as-of 2026-08-28 rather than assumed. A capitalised modal opening a sentence or a table cell reads as a month, which over-reports into a bucket a human reads and is the safe direction; seven of the 24 capitalised "May" lines are that shape. ALL-CAPS defeats case, so a digitless ALL-CAPS May date stays invisible, but all 34 ALL-CAPS "MAY" lines in the corpus are RFC-2119 modals and not one is a date, so reading that form as a month would cost 34 false candidates to buy a date form nobody writes. A digit still rescues "VERIFIED MAY 2026". Corpus figures are unmoved: 527 candidates, 499 parsed, 28 declined, 0 findings, and extract-breadcrumbs.sh inventories the same 517 stamp lines. The repo carries no digitless May stamp today, so this restores a detection the fix had silently dropped rather than changing what the tree reports. The suites' existing cross-script agreement assertion could not have caught this: it asserts both scripts return the same count over a shared fixture, and it passed while both were equally wrong. The new cases pin non-zero expected counts on each side. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
…asurement Three independent reviewers found that requiring a digit beside "may" made a digitless stamp vanish, and vanish upstream of the declined bucket rather than inside it, so it escaped both audit paths without appearing anywhere a human reads. The correction is in 3c538bc; this records what happened and what it cost. Two things worth keeping beyond the fix. The suites could not have caught it. They assert both scripts return the same count over a shared fixture, which passes when both are equally wrong, and both were. A cross-implementation agreement test detects divergence and is blind to a common error, and a shared definition is what makes a common error likely. The new cases pin a non-zero expected count on each side, so agreement is backed by a known answer instead of two implementations nodding at each other. The case signal was measured rather than assumed: over 1,352 files, 1,458 lines carry a lowercase modal "may", 24 carry a capital "May" of which about 17 are dates, and 34 carry ALL-CAPS "MAY" of which none is a date. That is what makes the capital usable and what rules out reading ALL-CAPS as a month. Both accepted costs are named and tested. Corpus figures are unchanged at 527 / 499 / 28 / 0: this tree carries no digitless-May stamp, so the regression was latent here and real in principle, which is why tests rather than a corpus count had to catch it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
|
Claude finished @kyle-sexton's task in 4m 3s —— View job Code review complete
Scope: this is a re-review at the new head ( The flagged regression is fixedTraced
On the corpus figures
No new correctness issues found. The two issues raised in the prior two rounds (the "13 vs 17 of 22" figure mismatch, and this digitless- |
|
Claude finished @kyle-sexton's task in 1m 0s —— View job Security review complete ✅
Skip gate not met — new code changed since the last review, so this ran a full pass on the delta. Last security-reviewed head:
Reviewed the new
No CRITICAL / IMPORTANT / SUGGESTION findings to report. The correctness question this delta raises (digitless-May handling) is outside this lane's scope — it's already being tracked in the code-review thread on this PR. |
…sified Three findings from the fresh-context verifier on the digitless-May fix. It passed the code and failed the state, on numbers rather than logic. The capitalised-May split was wrong. The comment and the changelog said seven of the 24 capital "May" lines are modals and about 17 are dates. Read line by line at 3c538bc it is ten modals and 14 dates. That figure sat in the paragraph headed "measured on this corpus, not assumed", one commit after a commit whose whole purpose was reconciling a figure to one measured number. Corrected in both places, with the commit attached. The "corpus effect: none" claim falsified itself one commit later. It is true at 3c538bc, where the tree carried no digitless-May stamp. But the paragraph explaining the fix quotes "Verified this May" inside a verified keyword window, in a file the corpus scans, so from a827aa5 onward the corpus does carry one: 529/499/30 post-fix against 528/499/29 pre-fix, an effect of +1 rather than none. Prose about the detector moved what the detector reports, for the fourth time on this branch. Every figure in the entry now carries the commit it was taken at, which is the rule the file already states and keeps relearning. Also records a latent under-report the verifier constructed: may_form() returns on the digit branch, so RSTART belongs to that match, and a digit-adjacent "may" beyond the window makes the caller reject before the in-window capital is consulted. Appending a stray "7 may" to a valid line removes its candidacy. No corpus line has that shape. Left rather than chased, and recorded at the rule: returning the leftmost of the two matches would fix it, while trying the capital branch first only mirrors the bug, so the obvious one-line swap is not a fix. Both scripts inherit it identically, so the cross-script agreement assertion is blind to it, exactly as it was to the regression this commit series started from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
|
Claude finished @kyle-sexton's task in 2m 43s —— View job Code review complete
Scope: re-review at the new head (
No executable awk/bash logic changed in this delta — The newly-documented edge case checks outTraced the claimed gap by hand against both scripts'
This matches the class of under-report the PR spent the last two rounds fixing, but it's correctly scoped as a documented, deliberately-unfixed latent case (no corpus line has the shape, and the comment explains why the obvious one-line reorder doesn't fix it — swapping branch order just moves which side of the bug you're on). No new test coverage was added for it, consistent with how it's framed: recorded at the rule, not shipped as a fix. Numbers check out internallyTraced the two corpus-measurement claims in
One minor observation, not flagged inline as it's below the high-signal bar for this lane: the No new findings to post inlineNothing in this delta rises to a blocking or flag-worthy issue — it's a self-consistent documentation correction that closes out the last round's regression discussion. Branch: |
|
Claude finished @kyle-sexton's task in 1m 7s —— View job Security review complete ✅
Skip gate hit — no new security-relevant surface since the last review. Last security-reviewed head:
No new executable logic, permissions, secrets, auth, injection surface, or GitHub Actions config changed in this delta. The prior review's findings (no CRITICAL/IMPORTANT/SUGGESTION on the No CRITICAL / IMPORTANT / SUGGESTION findings to report. |
…hape The code-review lane noted, below its flagging bar, that the a827aa5 figure gave three numbers where the 3c538bc figure beside it gave four, dropping findings. In an entry whose whole subject is figures that reproduce, two adjacent measurements should be diffable literally. Both now read candidates / parsed / declined / findings, and findings is 0 on both sides, which is the invariant the entry relies on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
|
Claude finished @kyle-sexton's task in 51s —— View job Code review complete
Scope: re-review at the new head (
No executable code, test, or other doc file changed in this delta. This closes the observation from the prior roundThe last review round noted (as a non-blocking observation, not a finding) that the Checked the arithmetic on the corrected figure, consistent with the surrounding prose (both figures were already validated in the prior round; this only adds the trailing
That's internally consistent with the entry's stated No new findingsThis is a self-contained prose correction with no executable surface and no new claims — nothing to flag inline. All prior rounds' findings (the "13 vs 17 of 22" figure mismatch, the invented |
2 participants
No linked issue
Summary
Rubric v3 shipped in #3471 with an obligation attached: its own rule blocks every precision figure, and with it every class's fix eligibility, until the golden set is re-scored against v3. That re-score is done here. It reproduces the recorded table exactly — and the run that produced it found that the rubric carries its own answer key, which is the more useful result.
Also closes the two design gaps the report-only sweep recorded on #3465, and fixes a false positive in the stamp detector that two independent sweep legs found.
Fix
The v3 re-score: 8 tp / 0 fp / 0 fn / 2 tn, no verdict moved
Ten cases, three judges each, thirty judges in independent OS processes — not sequential passes in one context. Cases relabelled so no directory name or path reached a judge. Each saw the candidate passage, the fetched source, the whole containing file and the rubric, per the v3 dispatch; none saw
expected.json, the fingerprint figures, or another judge's verdict. Every panel unanimous.c04is the only case whose attribution reaches grading, so it is the only one C3's stated scope could have moved. All three judges took the new test where v2 left it: the derivation is one lift inside otherwise-original material, so aSee alsobullet two sections below understates its scope and C3 passes.No class becomes fix-eligible, for the reason that was already there. Every class measures 1.00 against the 0.95 bar and every class sits below
min_n_per_class10 (n = 2, 5, 1, 2). The re-score lifts the rubric-version block; the class-size block stands, and is what keeps the #3465 sweep report-only.The rubric carried its own answer key
In #3471 I added a paragraph to
rubric.mdrecording the expected tally, the panel size, and an enumeration of which golden case turns on which criterion — including the one case the scope change exists to restate. The pipeline inlines the whole rubric into every judge prompt at the judgment step. All thirty judges read the prediction before grading.I wrote it so the figures would not sit under a cloud they did not deserve. Putting it in the file judges read is what made a blind measurement against that rubric impossible. Found by the re-score's own fresh-context verifier, which returned FAIL on the method while confirming the arithmetic; the run withdrew its blindness claim rather than defending it.
The verdict was tested against the leak rather than assumed safe.
c04was re-judged by a second three-judge panel against the same rubric with the preamble removed and every criterion, carve-out, scope sentence and worked example intact. All three returned STANDS on the same scope-mismatch reasoning. The tally stands; the blind-panel claim does not.A second leak of the same kind sits in the
c08/c09/c10sharedsource.md, and is deliberately not fixed here: that line is inside the text the fingerprint compares, so removing it moves the containment figures those cases record. The fix and a re-score are one atomic change; splitting them would ship a measurement that no longer reproduces from its own fixtures.The modal "may" is no longer read as a month name
mayis a month and an ordinary English modal, and both detectors matched it bare, so "the first read may raise a permission prompt" became a stamp candidate with an unparseable date and landed in the declined bucket — indistinguishable, to whoever adjudicates that bucket, from a real stamp the parser failed on. 17 of the 22 month-name declines carried the word.maynow needs a digit beside it. Three sites, not two: both detectors and the classifier incheck-stamps.sh, which a single-site fix would have missed. The other eleven months still match bare, because over-reporting into a bucket a human reads is the safe direction.Over 1,352 files: declines 45 → 28, month-name 22 → 5.
parsedunchanged at 499 andfindingsunchanged at 0 — the load-bearing numbers, since they say no real stamp was reclassified and none had been masked.Two design gaps from the sweep
A vendored-snapshot basis now has a rule: caps at
source-fetched-similar, recordssource.routewith each failed live fetch, and is never fix-eligible — fix eligibility rests on current upstream state, which a snapshot cannot establish. The searched-surfaces listing is recorded as prose-only and unenforced, becauseemit-findings.shhas no field for it; a run's listing is that run's own claim, never validation evidence.Verification
Four independent fresh-context verifiers, none judging its own work. Two returned FAIL and both were right:
main;git diff --diff-filter=Dis now empty and all seven files are present.mayfix's verifier passed the change, then caught that the changelog cited an invented exhibit (last_verified_2026_08, which exists nowhere in the repo) and supplied the real one, and that the fix had quietly falsified a comment in both scripts about window slack.Red-first reproduced verbatim (56/3 and 47/3 against pre-fix scripts, 59/0 and 50/0 after), test diffs additive-only (73 insertions, 0 deletions),
May 2026still detected in both forms in both scripts, mawk portability confirmed, shellcheck clean repo-wide.Limits recorded rather than left to be discovered. The re-score's panel was not blind, for the reason above. Four case bodies state their own intended answer and v3 requires judges to read the whole file.
c07'sexpected.jsondescribes two mutually exclusive routes; all three judges took the carve-out, which the rubric's order of evaluation requires, and neither the fixture nor the rubric was edited to match the run. A latent window-slack edge exists for the unboundedmayrule and is documented at the rule rather than fixed, since widening the slack would move a boundary two earlier commits tuned.Related
🤖 Generated with Claude Code
https://claude.ai/code/session_018tvKPLnWr7suxxAupU4p8A
Generated by Claude Code