Stamp audit runs and flag stale driftNotes (#341) - #352
Merged
Conversation
Both halves of #341 are about the same failure: a point-in-time snapshot presented as current state, so an agent picking it up "fixes" what is already fixed. 1. Freshness. spec/audit.py prints a run stamp (audit run <UTC> | hub <sha>) and, per repo, the exact commit it read (@ <branch>@<sha>), and closes with a re-run reminder. AUDIT.md section 8 requires anything derived from a run - a report, and especially an onboarding issue - to quote the stamp, and requires the agent picking it up to re-run the audit and act on the live result rather than the pasted findings. 2. driftNotes. A note records a *current* deviation; once resolved it is deleted, not left describing finished work. audit.py now flags a note asserting outstanding work (pending / not yet / missing / behind / ...) on a repo that otherwise audits clean - the exact pattern reconciled reactively on three repos (#329, #335, #348). The marker list is deliberately narrow so a note recording a permanent deviation ("no get-version-task; relies on validate-task") never trips it. Codified as setup.driftnotes.current. The new check immediately earned its keep: it caught a stale note I wrote myself for HomeAutomation-Config ("OWNER TASK OPEN: ... secrets not yet set") that the owner had since resolved and I never reconciled. Verified the secrets are present in both stores and removed the note - conformance being complete, it recorded no deviation at all. Fleet-wide the heuristic flagged exactly that one note, so it is low-noise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds explicit freshness signals to spec/audit.py output (run stamp + per-repo audited commit) and codifies a low-noise heuristic to flag stale registry.driftNotes that still claim outstanding work when a repo otherwise audits clean. This strengthens the audit/onboarding workflow by making audit-derived findings clearly attributable to a point-in-time snapshot and reducing “already-fixed” churn at issue pickup.
Changes:
- Stamp audit runs with UTC time + hub SHA, and print per-repo
@ <branch>@<sha>to attribute findings to an exact audited commit. - Add a drift finding when
driftNotesinclude “outstanding work” markers on a repo that otherwise audits clean; document the rule inAUDIT.mdandspec/project-types.json. - Remove a stale
driftNotesentry fromregistry/repos.jsonthat described already-resolved work.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| spec/project-types.json | Adds an intent check describing the new audit freshness/driftNotes behavior. |
| spec/audit.py | Implements run/repo stamping and adds the stale-driftNotes heuristic. |
| registry/repos.json | Removes a resolved/now-stale driftNote block for HomeAutomation-Config. |
| AUDIT.md | Documents the “stamp findings + re-run at pickup” requirement and driftNotes reconciliation expectation. |
…pilot #352) Three fixes, the first two real bugs introduced by this PR: 1. audit_repo's early gh-failure path still returned a bare findings list while the normal path returned (findings, sha), so unpacking it raised ValueError - which main's except Exception then reported INSTEAD of the original gh error, masking it. Returns a tuple now. Regression-tested by forcing the first gh call to fail: the 403 is preserved. 2. audited_sha fell back to (branch_main or branch_dev), so a repo whose ground truth is develop but whose develop is missing would stamp "develop@<mainSHA>" - misattributing every finding. It now reads the branch matching groundTruthBranch and stamps nothing when that branch is absent. 3. The driftNote marker match used plain substrings, missing "still."/"missing," at punctuation and risking hits inside longer words. Now a word-boundary regex; the "still " trailing-space hack is gone. Table-tested: 4 true positives and 3 negatives, including "missingness metric" and the permanent-deviation note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…opilot #352) The driftNotes comment ran six lines against the repo's comment-hygiene rule (one line default, a second only to carry a constraint); trimmed to two that keep the gating rationale and the false-positive guard. The clean-run message said "the full operational verdict", which collides with workflowModel: operational and misreads on a release repo. It now says "the full letter+intent verdict", naming the tiers AUDIT.md actually evaluates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
spec/audit.py:235
- audited_sha stamping only handles groundTruthBranch values 'develop' vs everything-else (treated as main). Since registry/repos.schema.json allows any string for groundTruthBranch, a non-standard branch name would produce a misleading stamp (e.g. audits ref 'master' but prints master@). Stamp should come from the actual groundTruthBranch ref to avoid misattributing findings.
# Stamp the commit actually read for the ground-truth branch. Never fall back to the other branch:
# a stamp naming develop while carrying main's sha would misattribute every finding.
ground_branch = branch_dev if ground == "develop" else branch_main
audited_sha = (ground_branch or {}).get("commit", {}).get("sha", "")
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #341. Both halves are the same failure mode: a point-in-time snapshot presented as current state, so an agent picking it up "fixes" what is already fixed.
1. Freshness signal on every run
spec/audit.pynow prints:@ <branch>@<sha>) - so a finding is attributable to a specific state.AUDIT.mdsection 8 now requires anything derived from a run - a report, and especially an onboarding/conformance issue - to quote the stamp, and requires the agent picking that issue up to re-run the audit and act on the live result, not the pasted findings. Findings are evidence for why the issue was filed, never a claim about current state. That is exactly the #16 failure: two of three findings were already resolved at pickup.2. driftNotes flagged when they outlive the deviation
A
driftNoterecords a current deviation; once resolved it is deleted, not left describing finished work.audit.pynow raises a drift finding when a repo audits clean but a note still asserts outstanding work (pending,not yet,missing,behind,owed,todo,still,absent).The marker list is deliberately narrow and the check only fires on an otherwise-clean repo, so a note recording a permanent deviation ("no get-version-task; relies on validate-task") never trips it. Codified as
setup.driftnotes.current.The check earned its keep immediately
It caught a stale note I wrote myself for HomeAutomation-Config yesterday -
"OWNER TASK OPEN: CODEGEN_APP_CLIENT_ID + CODEGEN_APP_PRIVATE_KEY not yet set in either secret store"- which the owner resolved (I verified the secrets and closed the downstream issue) but I never reconciled in the registry. Verified both secrets present in both stores, then removed the note entirely: with conformance complete it recorded no deviation at all.Noise check: run across the fleet, the heuristic flagged exactly that one note.
Validation
spec/validate.py, markdownlint, editorconfig-checker clean;audit.pyparses and runs; CRLF/LF preserved.This lands before the fleet-wide audit sweep deliberately - it determines whether the issues that sweep produces are accurate on arrival.
🤖 Generated with Claude Code