Stop the audit asserting a bypass list the config no longer manages - #551
Merged
Conversation
Removing bypass_actors from the ruleset payloads made every repo in the fleet report two ruleset DEFECTs. spec/audit.py compares the live ruleset against the payload over a fixed subset that included bypass_actors, so a payload that deliberately declares none against a live ruleset that has one is a normalized diff, and the audit called that divergence. The field is dropped from the compared subset rather than the payloads being changed back. Who may bypass a ruleset is a per-repository human decision taken in the UI, and configure.sh already treats it that way: apply writes the live list back unchanged, check reports it without asserting. The audit asserting it contradicted that directly. The defect was two tools comparing one field under opposite policies, not the field's value, so the comment says why the field is absent and what happened when it was not. The next person to add it back needs to know that. Found while reviewing the Blog registration in #547, whose report claimed nine drift and no defects. That was true when it was written and stopped being true an hour later when the payload change promoted, which is the same staleness the audit's own run-stamp rule exists to expose. Verified against two repos: Utilities and PlexCleaner both reported two ruleset DEFECTs before and neither does now. Utilities' remaining two findings are LETTER results for absent GOVERNANCE.md and OPERATIONS.md, which are the real propagation gap rather than this regression. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The explanatory comment landed as wrapped prose and tripped the gate this repo ships. Same content, one sentence per line, and it no longer opens on a lowercase path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a regression in spec/audit.py where ruleset audits started reporting fleet-wide DEFECTs after bypass_actors was removed from the ruleset payloads. The audit now aligns with the updated policy that bypass lists are UI-managed per repository rather than asserted by fleet config.
Changes:
- Remove
bypass_actorsfrom the ruleset normalization/comparison subset so audits no longer assert it. - Delete the now-dead bypass-actors sorting logic in
normalize_ruleset. - Add inline rationale explaining why
bypass_actorsis intentionally excluded from audit assertions.
The PENDING_MARKERS comment read 'assert work still outstanding', which is missing a verb and is the kind of line someone later greps for. It also carried a parenthetical example joined by a semicolon, which the character-set rules ban in agent-authored prose, so the example is now its own sentence without one. Pre-existing rather than introduced here, and adjacent to the change, so it is fixed in place rather than stepped around. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 3, 2026
ptr727
added a commit
that referenced
this pull request
Aug 3, 2026
Both findings from the review on promotion #555. A promotion's head is `develop`, so the fix lands here and #555 picks it up. ## The report predicted, and the prediction was wrong `reports/blog/audit.md` closed a paragraph with: > ...and the same four will appear against every fleet repo the moment that change promotes: Two things wrong with that, and the second is why it matters. **A report records what a run observed.** A prediction cannot be verified by re-running the audit, which is the one check the run-stamp discipline offers a reader. **And it turned out to be false.** The change promoted. The four findings were then traced to `spec/audit.py` asserting a `bypass_actors` list that `configure.sh` had deliberately stopped managing, and #551 removed that assertion. So they do not appear against every fleet repo. Verified: `spec/audit.py Blog` reports **0 defect/letter/error** on this branch. A prediction goes stale the moment reality diverges and nothing re-checks it. That is the same failure the registry's `driftNote` rules already name, arriving in a report instead — and it is the second instance today of a document asserting a future that stopped being true. The sentence now ends at the reason those findings were not counted, and hands off to the enumeration that follows it. ## The comment attached its clause to the wrong noun ```diff -# Phrases in a registry driftNote that assert work is still outstanding. +# Phrases that assert work is still outstanding, matched against a registry driftNote. ``` The subject is the plural *phrases*, so `assert` was correct, but `driftNote that assert` sits adjacent and reads as a disagreement. Reordered so the clause sits with what it describes. This is the second pass over these two lines today, the first having added a missing verb. ## Verification Diff-scoped `prose_lint` clean, editorconfig clean, CRLF preserved on the report, `spec/audit.py` parses, and the audit still reports Blog at zero findings. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ptr727
added a commit
that referenced
this pull request
Aug 3, 2026
…555) Three commits, and the first two are repairs to defects this hub shipped earlier today. Merged as a **merge commit**, never squashed. | Commit | PR | What | | --- | --- | --- | | `71b7691` | #547 | Catalogs Blog and records its first audit | | `82e3658` | #551 | Stops the audit asserting a bypass list the config no longer manages | | `fe01060` | #553 | Defines `walk` in the filter rather than calling it, for jq 1.5 | ## Why this one should not wait **`main` is currently wrong about every repository in the fleet.** The previous promotion took `bypass_actors` out of the ruleset payloads, but `spec/audit.py` compares live-against-payload over a subset that still included that field. A payload deliberately declaring no bypass, against a live ruleset that has one, reads as a normalized diff, so every repo audits as two ruleset DEFECTs: ```text == Utilities (csharp, nuget; release) @ main@e179288 == DEFECT ruleset: develop diverges from repo-config/develop.json (normalized diff) DEFECT ruleset: main diverges from repo-config/main.json (normalized diff) ``` Anyone auditing against `main` today gets a false failure on every repository. #551 removes the field from the compared subset, because the defect was **two tools comparing one field under opposite policies**, not the field's value: after the bypass change, `apply` writes the live list back unchanged and `check` reports it without asserting, so the audit asserting it contradicted the config directly. Verified on `develop`: Utilities and PlexCleaner went from two ruleset DEFECTs each to none, and Utilities' remaining two findings are LETTER results for absent `GOVERNANCE.md` and `OPERATIONS.md`, which is the real propagation gap. ## The second repair #553 closes #549. The payload-driven comparison was built on `walk/1`, which arrived in **jq 1.6**. On jq 1.5 the filter does not degrade, it fails to compile, so `check_ruleset` would report drift on every parameterized rule it never compared. Reproduced on a genuine `jq-1.5-1-a5b5cbe` and confirmed fixed, with output byte-identical to jq 1.7. Both defects were found by review rather than by a gate: #551 while reviewing the Blog registration, #553 by a Copilot review on a downstream re-vendor of the same canonical. ## Blog #547 catalogs Blog, taking the registry to 22 repositories. Its committed report claims zero defects, which was true when written, briefly false while the audit regression stood, and is true again on this branch. Confirmed rather than assumed: `spec/audit.py Blog` reports **0 defect/letter/error** here. ## Verification `spec/validate.py` OK at 22 cataloged, diff-scoped `prose_lint` clean, editorconfig clean, live read-only `configure.sh check` against this repo passes, and the audit reports the corrected verdicts above.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Regression fix. #545 removed
bypass_actorsfrom the ruleset payloads, and every repo in the fleet has been reporting two ruleset DEFECTs since it promoted.What broke
spec/audit.pycompares the live ruleset against the payload over a fixed subset, and that subset includedbypass_actors. A payload that deliberately declares no bypass list, against a live ruleset that still has one, is a normalized diff — so the audit called it divergence:Reproduced on Utilities, PlexCleaner and Blog. It is every repo, because every live ruleset carries the entry the payloads stopped declaring.
The fix, and why it is this way round
The field is dropped from the compared subset rather than restored to the payloads.
Who may bypass a ruleset is a per-repository human decision taken in the UI, and
configure.shalready treats it that way after #545:applywrites the live list back unchanged,checkreports it without asserting. The audit asserting the same field contradicted that directly.The defect was two tools comparing one field under opposite policies, not the field's value. The comment says so and records what happened when it was not, because the next person to look at that list will otherwise put it back.
How it was found
While reviewing the Blog registration in #547. That report claims nine drift and no defects, which was true when it was written against hub
mainat3a7cc64, and stopped being true an hour later when #545 promoted. The PR body had even predicted it: "The same four will appear against every fleet repo when that promotes."It did. This is the same staleness the audit's own run-stamp rule exists to expose, arriving from the hub's side rather than the repo's.
Verification
GOVERNANCE.mdandOPERATIONS.md, which are the real propagation gap rather than this regression.spec/validate.pyOK, diff-scopedprose_lintclean, editorconfig clean, syntax checked.Follow-on
#547's
reports/blog/audit.mdshould be regenerated against this corrected tool before it merges, so the committed report matches what the audit now reports rather than what it reported an hour before the payload change landed.