Skip to content

Stamp audit runs and flag stale driftNotes (#341) - #352

Merged
ptr727 merged 3 commits into
developfrom
audit-freshness-driftnotes-341
Jul 18, 2026
Merged

Stamp audit runs and flag stale driftNotes (#341)#352
ptr727 merged 3 commits into
developfrom
audit-freshness-driftnotes-341

Conversation

@ptr727

@ptr727 ptr727 commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Fixes #341. Both halves are the same failure mode: a point-in-time snapshot presented as current state, so an agent picking it up "fixes" what is already fixed.

1. Freshness signal on every run

spec/audit.py now prints:

audit run 2026-07-18T15:11:28Z | hub bd48caf

== HomeAutomation-Config (source-only; operational) @ main@6fadcbe ==
  clean (deterministic checks; the full operational verdict is AUDIT.md's)
...
Findings are a point-in-time snapshot: re-run this audit before acting on them, and quote the
run stamp above in any issue derived from it (AUDIT.md section 8).
  • a run stamp (UTC + the hub commit the ground truth came from), and
  • per repo, the exact commit read (@ <branch>@<sha>) - so a finding is attributable to a specific state.

AUDIT.md section 8 now requires anything derived from a run - a report, and especially an onboarding/conformance issue - to quote the stamp, and requires the agent picking that issue up to re-run the audit and act on the live result, not the pasted findings. Findings are evidence for why the issue was filed, never a claim about current state. That is exactly the #16 failure: two of three findings were already resolved at pickup.

2. driftNotes flagged when they outlive the deviation

A driftNote records a current deviation; once resolved it is deleted, not left describing finished work. audit.py now raises a drift finding when a repo audits clean but a note still asserts outstanding work (pending, not yet, missing, behind, owed, todo, still, absent).

The marker list is deliberately narrow and the check only fires on an otherwise-clean repo, so a note recording a permanent deviation ("no get-version-task; relies on validate-task") never trips it. Codified as setup.driftnotes.current.

The check earned its keep immediately

It caught a stale note I wrote myself for HomeAutomation-Config yesterday - "OWNER TASK OPEN: CODEGEN_APP_CLIENT_ID + CODEGEN_APP_PRIVATE_KEY not yet set in either secret store" - which the owner resolved (I verified the secrets and closed the downstream issue) but I never reconciled in the registry. Verified both secrets present in both stores, then removed the note entirely: with conformance complete it recorded no deviation at all.

Noise check: run across the fleet, the heuristic flagged exactly that one note.

Validation

spec/validate.py, markdownlint, editorconfig-checker clean; audit.py parses and runs; CRLF/LF preserved.

This lands before the fleet-wide audit sweep deliberately - it determines whether the issues that sweep produces are accurate on arrival.

🤖 Generated with Claude Code

Both halves of #341 are about the same failure: a point-in-time snapshot
presented as current state, so an agent picking it up "fixes" what is
already fixed.

1. Freshness. spec/audit.py prints a run stamp (audit run <UTC> | hub
   <sha>) and, per repo, the exact commit it read (@ <branch>@<sha>), and
   closes with a re-run reminder. AUDIT.md section 8 requires anything
   derived from a run - a report, and especially an onboarding issue - to
   quote the stamp, and requires the agent picking it up to re-run the
   audit and act on the live result rather than the pasted findings.

2. driftNotes. A note records a *current* deviation; once resolved it is
   deleted, not left describing finished work. audit.py now flags a note
   asserting outstanding work (pending / not yet / missing / behind /
   ...) on a repo that otherwise audits clean - the exact pattern
   reconciled reactively on three repos (#329, #335, #348). The marker
   list is deliberately narrow so a note recording a permanent deviation
   ("no get-version-task; relies on validate-task") never trips it.
   Codified as setup.driftnotes.current.

The new check immediately earned its keep: it caught a stale note I wrote
myself for HomeAutomation-Config ("OWNER TASK OPEN: ... secrets not yet
set") that the owner had since resolved and I never reconciled. Verified
the secrets are present in both stores and removed the note - conformance
being complete, it recorded no deviation at all. Fleet-wide the heuristic
flagged exactly that one note, so it is low-noise.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 18, 2026 15:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds explicit freshness signals to spec/audit.py output (run stamp + per-repo audited commit) and codifies a low-noise heuristic to flag stale registry.driftNotes that still claim outstanding work when a repo otherwise audits clean. This strengthens the audit/onboarding workflow by making audit-derived findings clearly attributable to a point-in-time snapshot and reducing “already-fixed” churn at issue pickup.

Changes:

  • Stamp audit runs with UTC time + hub SHA, and print per-repo @ <branch>@<sha> to attribute findings to an exact audited commit.
  • Add a drift finding when driftNotes include “outstanding work” markers on a repo that otherwise audits clean; document the rule in AUDIT.md and spec/project-types.json.
  • Remove a stale driftNotes entry from registry/repos.json that described already-resolved work.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
spec/project-types.json Adds an intent check describing the new audit freshness/driftNotes behavior.
spec/audit.py Implements run/repo stamping and adds the stale-driftNotes heuristic.
registry/repos.json Removes a resolved/now-stale driftNote block for HomeAutomation-Config.
AUDIT.md Documents the “stamp findings + re-run at pickup” requirement and driftNotes reconciliation expectation.

Comment thread spec/audit.py
Comment thread spec/audit.py Outdated
Comment thread spec/audit.py
…pilot #352)

Three fixes, the first two real bugs introduced by this PR:

1. audit_repo's early gh-failure path still returned a bare findings
   list while the normal path returned (findings, sha), so unpacking it
   raised ValueError - which main's except Exception then reported
   INSTEAD of the original gh error, masking it. Returns a tuple now.
   Regression-tested by forcing the first gh call to fail: the 403 is
   preserved.

2. audited_sha fell back to (branch_main or branch_dev), so a repo whose
   ground truth is develop but whose develop is missing would stamp
   "develop@<mainSHA>" - misattributing every finding. It now reads the
   branch matching groundTruthBranch and stamps nothing when that branch
   is absent.

3. The driftNote marker match used plain substrings, missing
   "still."/"missing," at punctuation and risking hits inside longer
   words. Now a word-boundary regex; the "still " trailing-space hack is
   gone. Table-tested: 4 true positives and 3 negatives, including
   "missingness metric" and the permanent-deviation note.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 18, 2026 15:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

Comment thread spec/audit.py Outdated
Comment thread spec/audit.py Outdated
…opilot #352)

The driftNotes comment ran six lines against the repo's comment-hygiene
rule (one line default, a second only to carry a constraint); trimmed to
two that keep the gating rationale and the false-positive guard.

The clean-run message said "the full operational verdict", which collides
with workflowModel: operational and misreads on a release repo. It now
says "the full letter+intent verdict", naming the tiers AUDIT.md actually
evaluates.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 18, 2026 15:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

spec/audit.py:235

  • audited_sha stamping only handles groundTruthBranch values 'develop' vs everything-else (treated as main). Since registry/repos.schema.json allows any string for groundTruthBranch, a non-standard branch name would produce a misleading stamp (e.g. audits ref 'master' but prints master@). Stamp should come from the actual groundTruthBranch ref to avoid misattributing findings.
    # Stamp the commit actually read for the ground-truth branch. Never fall back to the other branch:
    # a stamp naming develop while carrying main's sha would misattribute every finding.
    ground_branch = branch_dev if ground == "develop" else branch_main
    audited_sha = (ground_branch or {}).get("commit", {}).get("sha", "")

@ptr727
ptr727 merged commit 7a3905b into develop Jul 18, 2026
7 checks passed
@ptr727
ptr727 deleted the audit-freshness-driftnotes-341 branch July 18, 2026 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants