Add spec/audit.py: on-demand live fleet audit (roadmap #310 slice D) - #324
Merged
Conversation
Mechanizes the deterministic subset of AUDIT.md, run from the hub against every cataloged registry repo (or named ones): general settings vs repo-config/settings.json, rulesets vs the model's payloads (normalized), secret names vs spec/secrets.json (required/forbidden/stale), baseline and per-type file presence on the ground-truth branch (spec/files.json, model-aware develop variant), and branch facts (main/develop existence, registry hasDevelop vs reality, content-divergence via tree compare - commit counts mislead under merge-commit promotions). Read-only; owner-initiated (onboarding, suspected drift, pre-change); exits non-zero on defect/letter findings. LF-pinned like validate.py; AUDIT.md section 0 points at it. Live-tested: Financial-Modeling reports exactly its known standup gaps; ProjectTemplate/PlexCleaner/HomeAutomation-Config exercised the release, docker, and operational paths. Refs #310. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an on-demand, local gh-authenticated Python runner to mechanize the deterministic (read-only) subset of AUDIT.md across the fleet, and wires it into existing audit documentation and line-ending governance.
Changes:
- Add
spec/audit.pyto audit live repo settings, rulesets, secret names, baseline file presence, and main/develop branch facts against hub manifests. - Update
AUDIT.mdto reference the new on-demand audit runner. - Pin
spec/audit.pyto LF via.gitattributesand.editorconfig(matching existing shebang-executed Python scripts).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| spec/audit.py | New CLI runner that performs the deterministic subset of the fleet audit via gh api. |
| AUDIT.md | Documents the new runner as the mechanized deterministic subset of the audit. |
| .gitattributes | Pins spec/audit.py to LF to keep shebang execution safe/correct. |
| .editorconfig | Pins LF for spec/audit.py alongside spec/validate.py. |
The docstring and summary now name the ERROR kind (hard failure), and a finding-free repo prints a neutral "clean (deterministic checks)" - the full operational verdict belongs to AUDIT.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Secrets audit honors registry requiredSecrets[] (STANDUP.md: requiredSecrets plus the implicit baseline) - required in the actions store, never stale. - main() isolates per-repo failures: a gh/JSON error mid-audit becomes that repo's ERROR finding instead of aborting the sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ptr727
added a commit
that referenced
this pull request
Jul 16, 2026
#326) Promotes two squashed increments: - **#323** (#322 incorporation) - publisher TOCTOU pin (checkout the dispatch-time commit), .github/ wording, adapted self-audit carry blessed as the standard downstream shape, tasks-snippet default build/test groups. - **#324** (#310 slice D) - spec/audit.py, the on-demand live fleet audit: settings/rulesets/secret-names/file-presence/branch-facts vs the ground truth, per-repo error isolation, tree-based divergence detection. Owner-initiated by design. The baseline App-pair requirement stays fleet-wide (owner-confirmed: the codegen App merges Dependabot PRs on every tier, operational included). Refs #310 (the cold-start self-test matrix remains the open acceptance item). Closes #322. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "runner next" half of the #310 roadmap decision (docs + matrix landed in #311). Scope per owner direction: on-demand only, owner-initiated (onboarding, suspected drift, deliberate changes) - no scheduled workflow, no App-token machinery; it runs under the local
ghauth.What it does
python3 spec/audit.py [RepoName ...](default: all 21 cataloged repos) executes the deterministic subset ofAUDIT.mdper repo, read-only:repo-config/settings.json(+ computedhas_discussions/default_branch)operationalrepos compareoperational/develop.json)spec/secrets.json: baseline + per-mechanism required in the right stores, forbidden anywhere, unclaimed flagged as stalespec/files.json(model-awaredeveloppayload path)hasDevelopvs reality, and content divergence via tree compare - commit counts mislead under merge-commit promotions, so only adevelop...maindiff with changed files (main carries content develop lacks) reportsFindings are typed
DEFECT/LETTER(file absent, intent unverified per AUDIT.md section 7) /DRIFT; exit is non-zero on defect/letter.AUDIT.mdsection 0 references the runner.Live-tested (read-only)
WORKFLOW.md,version.json,repo-config/*,dependabot.yml) absent onmainpending PR Improve setup instructions and documentation consistency #15; settings/rulesets/secrets clean (as applied).NINJA_API_KEY/NUGET_API_KEY; HomeAutomation-Config missing the baseline App pair; PlexCleaner missing the repo-config carry) - reported to the maintainer separately, not acted on.Refs #310 (the cold-start self-test matrix remains the roadmap's open acceptance item).
🤖 Generated with Claude Code