From 4143519ee8796fb74e48dce66e22d3a4a612c629 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 22 Aug 2026 08:42:35 -0700 Subject: [PATCH 1/3] Add a Fleet Membership Check to spec/audit.py (#909) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary - Every hub tool (spec/audit.py, spec/validate.py, spec/fidelity_honesty.py, spec/workflow_reuse.py) iterated registry/repos.json and never checked whether it agreed with what actually exists on GitHub, so a repo that never got an entry was invisible to all of them. Blog operated for two days undetected this way, and DiskSpeedTest, GEM-Echo-Server, and GoogleTo1Password (all archived) were still missing from the registry. - spec/audit.py: new owner_repos()/membership_findings(), run once on a full sweep (skipped on a name-filtered or --issue run). Lists every non-fork repo the registry owner has on GitHub, reports one absent from registry/repos.json as a DEFECT, and reconciles a registry status: "archived" entry against GitHub's own archived flag as a DRIFT in either direction. Guards against querying the wrong account by comparing gh's authenticated login to the registry owner first. - registry/repos.schema.json + spec/validate.py: extended status to cataloged | backlog | archived | excluded. An excluded entry now requires a non-empty exclusionReason, so a deliberate decision not to audit a repo stays visible instead of reading as an oversight. - registry/repos.json: added status: "archived" entries for the three missing repos, so the new check is green on merge. - AUDIT.md, STANDUP.md, GOVERNANCE.md, README.md: documented the check, the archived/excluded statuses, and where a MISSING finding should send an agent (STANDUP.md). - TODO.md: retired the "Registry Membership Coverage" tracker entry, its open questions settled by the design above. ## Verification - `python3 spec/audit.py --selftest`: SELFTEST PASS, including 7 new cases covering owner_repos() pagination/fork-filtering and membership_findings()'s four finding shapes - `python3 -c "...membership_findings(...)..."` against the live registry: 0 findings (confirms the three archived stub entries close the gap the issue reported) - `python3 spec/validate.py`: OK, 22 cataloged, 0 backlog, 3 archived, 0 excluded - `python3 scripts/prose_lint.py`: 0 issues - `python3 scripts/repo_gate.py --check {eol,eol-coverage,sha-pin}`: 0 issues each - `python3 -m unittest discover -s scripts/tests`: 765 tests, OK - `ruff check` / `ruff format --check` on spec/audit.py, spec/validate.py: clean - `mypy spec/audit.py spec/validate.py`: no issues (pyright reports 4 pre-existing errors elsewhere in both files, unrelated to this diff) Fixes #550. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Summary by CodeRabbit - **New Features** - Added fleet-wide repository membership auditing to identify unregistered repositories and archive-status mismatches. - Added `archived` and `excluded` repository statuses, including required exclusion reasons. - Registered three archived repositories. - **Bug Fixes** - Validation now handles archived and excluded repositories appropriately and reports their totals. - Audits skip archived or excluded repositories when applicable. - **Documentation** - Updated governance, audit, README, and standup guidance for repository registration and status rules. --- AUDIT.md | 8 +- GOVERNANCE.md | 1 + README.md | 2 +- STANDUP.md | 2 + TODO.md | 18 -- registry/repos.json | 18 ++ registry/repos.schema.json | 12 +- spec/audit.py | 345 ++++++++++++++++++++++++++++++++++++- spec/validate.py | 78 +++++++-- 9 files changed, 442 insertions(+), 42 deletions(-) diff --git a/AUDIT.md b/AUDIT.md index 94f1da94..926db34b 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -9,7 +9,8 @@ The verdict vocabulary is [`WORKFLOW.md`][workflow]'s: **operational / not opera ```mermaid flowchart TD - s0["0: has the repo been stood up? if not, STANDUP.md"] --> s1["1: scope, ground-truth branch (main)"] + s0m["0m: fleet membership, every owned non-fork repo has a registry entry"] --> s0["0: has the repo been stood up? if not, STANDUP.md"] + s0 --> s1["1: scope, ground-truth branch (main)"] s1 --> s2["2: resolve the repo's type(s)"] s2 --> s3["3: applicability gate, per check"] s3 --> s4["4: per-dimension checks, letter and intent"] @@ -28,6 +29,7 @@ flowchart TD This audit is not occasional. Run it whenever you **create, adopt, or materially change** a fleet repo, and on demand for any known repo: +- **A full sweep opens with a fleet membership check, not a per-repo one.** `spec/audit.py`, run with no repo names, first lists every non-fork repository the registry `owner` actually owns on GitHub and diffs it against `registry/repos.json`. A repo that exists but carries no entry is invisible to every other check in this file, since all of them iterate the registry and never look past it, so this is the only place that gap is caught (ptr727/ProjectTemplate#550). The check also reconciles one field: a registry `status: "archived"` must agree with GitHub's own archived flag, in either direction. A name-filtered run or `--issue` skips it, since those are scoped to repos already known to the registry. Run this as `gh auth login` for the owner's own account: a fine-grained PAT scoped to "selected repositories" returns an incomplete list with no error, so the sweep would read clean while some repos were never inspected. - **Onboarding a repo is complete only when it either passes this audit** (operational on every applicable check) **or carries a committed `reports//audit.md` plus a tracking issue** enumerating every residual delta. A repo that is partially set up but never audited is itself a **defect**, the exact state this process prevents. The create-to-conformance counterpart is [`STANDUP.md`][standup]. Because both read the same manifests, a repo stood up by that file passes this audit by construction. - **Touching a repo** (any conformance-affecting change) ends by re-running the applicable checks and **reconciling the registry entry to reality**: `status`, `types`, `releaseTrigger`, `workflowModel`, `driftNotes`. The registry records reality, not intent. [`spec/validate.py`][validate] proves the catalog is self-consistent, not that it matches the live repo. Closing that gap is this audit's job. The deterministic subset (settings, rulesets, secret names, file presence, per-scope Markdown section presence, workflow interface conformance, verbatim content, hub-hosted files a repo carries, branch facts) is mechanized in [`spec/audit.py`][audit-runner]: owner-initiated, run on demand when onboarding a repo, on suspected drift, or before fleet-wide changes. A required section missing from a carried Markdown file is a **drift finding**, not a letter, because a heading rename reads as missing and equivalence is judged by hand. A carried `interface` workflow (spec/fidelity-model.md) is checked by name and wiring (required jobs, the ruleset-bound check name, the artifact-name handoff, and the forbidden `artifact-ids:` fork), all at **drift**, since the body is owned and a rename is a hint to verify. A carried `verbatim` unit, whether a whole file (`.markdownlint-cli2.jsonc`) or a canonical workflow job region (the `github-release` job), is content-hashed against the hub's canonical after line-ending normalization. A mismatch is classified **stale** (matches a past hub revision, re-vendor) or **modified** (matches none, the repo changed fixed content), both at **drift**, since equivalence is intent-governed and a byte diff is a hint to review. A carried `intent` unit gets one advisory beyond presence, a last-modified comparison: a hub canonical changing after the copy's own last commit marks the copy as possibly trailing, at **drift**, a hint rather than proof, since a copy touched without reconciling reads current and content is never judged. @@ -47,7 +49,9 @@ This holds for **both workflow models**. An `operational` repo commits directly ## 2. Resolve the Repo's Type(s) -Look up the repo in [`registry/repos.json`][repos] and read its `types[]`. If the entry is `classificationPending` (a backlog repo), classify it from the tree and propose a registry update: +Look up the repo in [`registry/repos.json`][repos]. An entry with status `archived` or `excluded` is out of scope for the rest of this procedure, so stop here rather than proceeding to section 3. `archived` means GitHub itself reports the repo archived, so no further conformance work applies. `excluded` means a maintainer decision took it out of audit scope, recorded in the entry's `exclusionReason`. Both still carry a registry entry precisely so the decision stays visible, per section 0's membership check, rather than the repo reading as an oversight. + +Otherwise read its `types[]`. If the entry is `classificationPending` (a backlog repo), classify it from the tree and propose a registry update: - `*.csproj` / `*.slnx` -> `csharp`, a `dotnet nuget push` workflow -> `nuget`, a `System.CommandLine` console -> `console`. - `pyproject.toml` / `setup.py` -> `python`, a `pypa/gh-action-pypi-publish` workflow -> `pypi`. diff --git a/GOVERNANCE.md b/GOVERNANCE.md index fa8bb68f..75edfc1e 100644 --- a/GOVERNANCE.md +++ b/GOVERNANCE.md @@ -104,6 +104,7 @@ Every fleet repo is a standard-style repo the hub audits **downward** against it - **Definition of done is the audit.** Onboarding or materially changing a repo is complete only when it passes [`AUDIT.md`](./AUDIT.md) for its type, **or** carries a committed `reports//audit.md` plus a tracking issue for the residual deltas. Do not leave a repo partially stood up and unrecorded, which is itself a defect. Stand a repo (or a new type) up with `STANDUP.md`, and verify it with [`AUDIT.md`](./AUDIT.md). The two share the same manifests, so a repo stood up correctly passes the audit by construction. A repo that is stood up already and has fallen behind the hub is brought back into line with `RESYNC.md`, which is the third procedure and the one a request to sync a repo with the hub means. It exists because a findings list does not state the order its remedies apply in, and one of them deletes a file rather than updating it. - **The registry is ground truth about reality, not intent.** After any conformance change, reconcile the repo's `registry/repos.json` entry (`status`, `types`, `releaseTrigger`, `workflowModel`, `driftNotes`) to what the repo actually is. `spec/validate.py` proves the catalog is self-consistent. It does **not** prove the catalog matches the live repo. A `validate.py`-clean entry can still be false, and a stale entry is a finding. +- **Every owned, non-fork repo carries a registry entry, full stop.** `spec/audit.py`'s fleet membership check (a full sweep, not a per-repo one) lists the owner's actual GitHub repos and flags any absent from the registry as a `DEFECT`, since every other tool here only ever iterates the registry and cannot see past it (ptr727/ProjectTemplate#550). An archived repo still gets an entry, `status: "archived"`, and an entry a maintainer has decided not to audit is `status: "excluded"` with a required `exclusionReason`. Both keep the decision visible rather than letting the repo disappear from the catalog. - **Prove the docs suffice.** The onboarding docs are sufficient only when a context-free agent stands each supported repo shape (project type(s) plus workflow model) up from them alone. The cold-start self-test in `STANDUP.md` and the conformance matrix track that. A shape that cannot be stood up cold is a documentation defect, fixed in the hub, not worked around per repo. ## Hub-Hosted Tooling diff --git a/README.md b/README.md index cb1b8f5f..ab0b8676 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,7 @@ This repo is the single home for those rules, a machine-readable spec they are c - **[AUDIT.md][audit]** - how an agent audits a repository against the spec and reports drift. - **[RESYNC.md][resync]** - how an agent brings an already-stood-up repository back into line, in the order the remedies require. - **[spec/][spec]** - the machine-readable ground truth: project-type requirements, the file/section baseline, required/forbidden secrets, the host tool contract and its version floors, and the preferred README structure. -- **[registry/repos.json][repos]** - the fleet registry: every project, its type(s), publish mechanism, and status (cataloged or standardization backlog). +- **[registry/repos.json][repos]** - the fleet registry: every project, its type(s), publish mechanism, and status (cataloged, standardization backlog, archived, or excluded). `spec/audit.py` checks the registry against the owner's actual GitHub repos, not just against itself, so a repo can't go missing from it unnoticed. - **[repo-config/][repo-config]** - hub-only branch rulesets, fleet settings, the apply script, and the GitHub setup reference (kept out of `.github/`, which is Actions-owned). - **[host-setup/][host-setup-dir]** - the host guardrail kit, which is per machine rather than per repository. - **[catalog/][catalog]** - reusable reference snippets (workflow tasks, config exemplars, devcontainers) the audit compares implementations against. diff --git a/STANDUP.md b/STANDUP.md index 33bbb549..4dbd33c4 100644 --- a/STANDUP.md +++ b/STANDUP.md @@ -115,6 +115,8 @@ The rename runs unconditionally rather than behind a test of `init.defaultBranch ## 1. Classify and Catalog +A repo lacking any entry is exactly what [`AUDIT.md`][audit] section 0's fleet membership check surfaces on a full sweep, a `DEFECT` naming the repo by its GitHub `full_name`. That finding is what should send an agent here in the first place for a repo nobody has stood up yet, rather than the omission sitting unnoticed (ptr727/ProjectTemplate#550). + Resolve the repo's type(s) with the [`AUDIT.md`][audit] section 2 detection rules, then write or repair its [`registry/repos.json`][repos] entry: `status`, `types[]`, `groundTruthBranch`, `hasDevelop`, `publish[]`, `requiredSecrets[]`, `consumerModel`, `releaseTrigger`, `workflowModel` (omit to take the `release` default), `configLayout`, and `driftNotes` that describe what the repo **actually is**. Run [`spec/validate.py`][validate] to confirm it classifies cleanly. The registry is ground truth about reality, not intent, and a `validate.py`-clean entry is still false if it disagrees with the live repo. ## 1A. Carry the Instruction Set, Before Authoring Anything diff --git a/TODO.md b/TODO.md index d0919036..0ea1cb91 100644 --- a/TODO.md +++ b/TODO.md @@ -52,23 +52,6 @@ One pull request moving the canonical short description into declared data, so e - **Settled** - The field is optional at first so the audit falls back to the README intro while repos adopt it, and it needs a schema entry because `registry/repos.schema.json` sets `additionalProperties: false`. - **Settled** - The ask on the Docker repos meanwhile is only that the parsing step is not propagated further. -### Registry Membership Coverage - -One pull request asking the inverse question the fleet tools never ask, whether a repository that exists has a registry entry, since every tool iterates the registry and an omission at standup is permanent and silent. - -**State** `decision`. **Touches** [`spec/audit.py`][audit], [`registry/repos.json`][repos] and its schema, and [`STANDUP.md`][standup]. **Cost** one hub edit, hub-only. - -- **Report a non-fork repository under the owner that has no registry entry.** The reports read as complete while under-counting today. - - **Blocked by** - Nothing. - - **Issue** - [#550][issue-550], which carries the four repos the comparison found. - - **Checked** - `develop` at `362aec8`, per the issue, and unverified since. - - **Open** - How a deliberate exclusion is recorded, since without one the check becomes a permanent four-line complaint people learn to scroll past, and the candidates are a third `status` value or a separate list carrying a reason per entry. - - **Open** - Where the check runs, since neither `validate.py` in CI nor an owner-initiated audit catches an omission at the moment it is made, which is the standup itself and the moment the fix costs one line. - - **Settled** - The consequence is worse than a gap, because the reports are confidently wrong rather than silent: [reports/divergences.md][divergences-report] counted 19 repos owing `AGENTS.md` "Fleet Bootstrap" when the real number was 20. - - **Settled** - The procedure is not the gap, since [`STANDUP.md`][standup] section 1A already says to write the entry and names every field, and nothing verifies it happened. - - **Settled** - The reason matters more than the mechanism, since an unexplained exclusion is the same silent omission in a different file. - - **Settled** - Private repositories are outside the public listing the issue used, so the true count is a floor rather than a total. - ### Reducing the Carried Surface Further One pull request measuring the remaining carried surface against the carry-versus-reach test and moving whatever qualifies, now that the model is settled rather than open. @@ -535,7 +518,6 @@ Regenerate [reports/divergences.md][divergences-report] before using it as the w [issue-509]: https://github.com/ptr727/ProjectTemplate/issues/509 [issue-521]: https://github.com/ptr727/ProjectTemplate/issues/521 [issue-523]: https://github.com/ptr727/ProjectTemplate/issues/523 -[issue-550]: https://github.com/ptr727/ProjectTemplate/issues/550 [issue-558]: https://github.com/ptr727/ProjectTemplate/issues/558 [issue-578]: https://github.com/ptr727/ProjectTemplate/issues/578 [issue-580]: https://github.com/ptr727/ProjectTemplate/issues/580 diff --git a/registry/repos.json b/registry/repos.json index 4927e659..9347dfbb 100644 --- a/registry/repos.json +++ b/registry/repos.json @@ -320,6 +320,24 @@ "releaseTrigger": "dispatch-only", "configLayout": { "rulesetsDir": "repo-config", "pythonConfig": null }, "driftNotes": ["Hugo static site migrated off WordPress.com, stood up 2026-08-01; release model with a dispatch-only publisher that cuts the tag and a source archive.", "lineEndings lf on a release repo, where the rule grants the native-platform default to operational repos only: every consumer is Linux (the Hugo build, the Caddy container, and the rsync deploy).", "content/ is an imported WordPress archive, so the prose, spelling, and style gates are scoped to exclude it.", "Deploy credentials are per-environment GitHub Environment secrets and variables, which spec/secrets.json has no vocabulary for, so requiredSecrets stays empty and the names are declared in this repo's own secrets.json environments block; listing them here would make the audit demand them in the repository actions store."] + }, + { + "name": "DiskSpeedTest", + "url": "https://github.com/ptr727/DiskSpeedTest", + "status": "archived", + "driftNotes": ["Archived on GitHub. Found missing from the registry by the fleet membership check (ptr727/ProjectTemplate#550). No further conformance work is expected while archived."] + }, + { + "name": "GEM-Echo-Server", + "url": "https://github.com/ptr727/GEM-Echo-Server", + "status": "archived", + "driftNotes": ["Archived on GitHub. Found missing from the registry by the fleet membership check (ptr727/ProjectTemplate#550). No further conformance work is expected while archived."] + }, + { + "name": "GoogleTo1Password", + "url": "https://github.com/ptr727/GoogleTo1Password", + "status": "archived", + "driftNotes": ["Archived on GitHub. Found missing from the registry by the fleet membership check (ptr727/ProjectTemplate#550). No further conformance work is expected while archived."] } ] } diff --git a/registry/repos.schema.json b/registry/repos.schema.json index 92164cfd..be6df3a5 100644 --- a/registry/repos.schema.json +++ b/registry/repos.schema.json @@ -36,13 +36,19 @@ "$comment": "An operational repo must declare its line endings (release repos use the fleet LF default).", "if": { "properties": { "workflowModel": { "const": "operational" } }, "required": ["workflowModel"] }, "then": { "required": ["lineEndings"] } + }, + { + "$comment": "An excluded repo records the deliberate decision that put it out of audit scope, not just the fact.", + "if": { "properties": { "status": { "const": "excluded" } }, "required": ["status"] }, + "then": { "required": ["exclusionReason"] } } ], "additionalProperties": false, "properties": { - "name": { "type": "string" }, - "url": { "type": "string", "format": "uri" }, - "status": { "enum": ["cataloged", "backlog"] }, + "name": { "type": "string", "minLength": 1 }, + "url": { "type": "string", "format": "uri", "minLength": 1 }, + "status": { "enum": ["cataloged", "backlog", "archived", "excluded"] }, + "exclusionReason": { "type": "string", "minLength": 1, "pattern": "\\S" }, "types": { "type": "array", "items": { "type": "string" } }, "profiles": { "type": "object", "additionalProperties": { "enum": ["lint-only", "build"] } }, "classificationPending": { "type": "boolean" }, diff --git a/spec/audit.py b/spec/audit.py index 19c7e300..48e9a387 100755 --- a/spec/audit.py +++ b/spec/audit.py @@ -8,7 +8,9 @@ hub-hosted files a repo carries and should delete (git-tracked here and undeclared in the manifest, triaged by spec/divergences.json), intent-staleness advisories (a carried intent file whose hub canonical changed after the copy last did), and branch-model facts (main/develop existence, develop -behind main). Owner-initiated: run it when +behind main), and a fleet-wide membership check (every non-fork repo the registry owner has on +GitHub carries a registry/repos.json entry, and an 'archived' entry's status agrees with GitHub's +own archived flag (ptr727/ProjectTemplate#550). Owner-initiated: run it when onboarding a repo, when drift is suspected, or before fleet-wide changes. Read-only - it never modifies a target. @@ -213,6 +215,115 @@ def gh(path, ok404=False) -> Any: return json.loads(r.stdout) if r.stdout.strip() else None +def owner_repos(owner): + """Every non-fork repository the registry owner actually owns on GitHub, paginated by hand. + + /user/repos rather than /users/{owner}/repos: the latter returns only public repos unless gh + is authenticated as that exact user, which would silently under-count a private repo the same + way a missing registry entry does. Confirm the authenticated login matches owner first, so a + mismatch is a loud error rather than a quietly wrong (and possibly cross-account) result. + No --paginate, per the gh() docstring above. Page by hand until a page comes back short. + + This still assumes the credential gh runs as can see every repo the owner has: an ordinary + `gh auth login` session does, but a fine-grained PAT scoped to "selected repositories" would + return an incomplete list with no error and no signal in the response to detect that from, so + the sweep would read as clean while some repos were never inspected. Run this as the owner's + own account, not a repository-scoped credential. + """ + me = gh("user") + login = me.get("login") if isinstance(me, dict) else None + # GitHub logins are case-insensitive. + # Compare lowered, or a registry owner spelled with different casing than gh's own login reads as a mismatch when it is the same account. + if (login or "").lower() != owner.lower(): + raise RuntimeError( + f"gh is authenticated as '{login}', not registry owner '{owner}'. " + "The membership check would query the wrong account's repos." + ) + repos, page = [], 1 + while True: + batch = gh(f"user/repos?affiliation=owner&per_page=100&page={page}") + # An empty response body reads as None here, per the gh() docstring above. + # `or []` would silently take that for a short page and stop the sweep, reporting the fleet clean while the rest of the repos were never actually inspected. + # Fail loud instead, matching the ownership guard above. + if batch is None: + raise RuntimeError(f"GitHub returned an empty repository-list response (page {page})") + if not isinstance(batch, list) or not all(isinstance(r, dict) for r in batch): + raise RuntimeError(f"GitHub returned an unexpected repository-list shape (page {page})") + repos.extend(r for r in batch if not r.get("fork")) + if len(batch) < 100: + return repos + page += 1 + + +GITHUB_URL_RE = re.compile(r"^https://github\.com/([^/\s]+)/([^/\s]+?)(?:\.git)?/?$") + + +def repo_identity(url): + """owner/repo, lowercased, parsed from a github.com URL, or None if it does not parse. + + The canonical identity two repos are compared by, since a bare repo name collides across + owners and GitHub's own full_name field is already this exact shape (case-preserved). A + trailing .git is stripped so it still resolves to the same identity full_name would. + spec/validate.py rejects a url that fails to parse here at all, kept in sync with this regex. + """ + if not isinstance(url, str): + return None + m = GITHUB_URL_RE.match(url.strip()) + return f"{m.group(1)}/{m.group(2)}".lower() if m else None + + +def membership_findings(spec): + """Fleet-wide: every non-fork repo the owner has on GitHub must have a registry entry. + + This is the check ptr727/ProjectTemplate#550 asked for: nothing else in this file, or in + spec/validate.py, ever looks past the registry to what actually exists, so a repo that never + got an entry is invisible to every tool that reads it. The registry was treated as ground + truth about existence, not just about conformance. Ownership only: a fork, or a repo the + owner merely collaborates on, was never meant to carry a registry entry. + + A registry entry's status then says what, if anything, the rest of the audit owes the repo: + 'cataloged' is audited normally, 'backlog' awaits classification (existing behavior), and + 'archived'/'excluded' are known and out of scope by design: archived because GitHub itself + says the repo takes no further work, excluded because exclusionReason records a human + decision that it should not be audited. Both still require an entry, so the decision stays + visible in the catalog instead of the repo just disappearing from it. + """ + findings = [] + # A bare-name match would treat a same-named repo under a different owner as this one's entry. + # Keyed by owner/repo instead, the shape GitHub's own full_name field already carries. + # Two entries resolving to the same identity are a separate, validate.py-level defect. + registry_by_identity = {} + for r in spec["registry"]["repos"]: + identity = repo_identity(r.get("url")) if isinstance(r, dict) else None + if identity is not None: + registry_by_identity[identity] = r + for gh_repo in owner_repos(spec["registry"]["owner"]): + entry = registry_by_identity.get(str(gh_repo.get("full_name", "")).strip().lower()) + if entry is None: + findings.append( + ( + "DEFECT", + f"{gh_repo['full_name']}: exists on GitHub with no registry/repos.json entry", + ) + ) + continue + gh_archived = bool(gh_repo.get("archived")) + reg_archived = entry.get("status") == "archived" + if gh_archived and not reg_archived: + msg = ( + f"{entry['name']}: archived on GitHub but registry status is " + f"'{entry.get('status')}' (reconcile to 'archived')" + ) + findings.append(("DRIFT", msg)) + elif reg_archived and not gh_archived: + msg = ( + f"{entry['name']}: registry status is 'archived' but GitHub reports it " + "unarchived (reconcile status, or verify it was not unarchived by mistake)" + ) + findings.append(("DRIFT", msg)) + return findings + + def docker_hub_description(slug): """The Docker Hub short description for a repo, or None if the image is genuinely absent (HTTP 404). @@ -4095,6 +4206,217 @@ def _selftest(): " ok repo_tree: a truncated tree and a missing tree sha both return None, and a whole one drops non-blobs" ) + id_cases = [ + ("https://github.com/owner/Repo", "owner/repo"), + ("https://github.com/owner/Repo/", "owner/repo"), + (" https://github.com/owner/Repo ", "owner/repo"), + ("https://github.com/owner/Repo.git", "owner/repo"), + ("https://github.com/owner/Repo.github", "owner/repo.github"), + ("http://github.com/owner/Repo", None), + ("https://gitlab.com/owner/Repo", None), + ("not a url", None), + (None, None), + (42, None), + ] + id_ok = all(repo_identity(url) == want for url, want in id_cases) + if id_ok: + print( + " ok repo_identity: parses owner/repo from a github.com URL, lowercased and trimmed" + ) + else: + ok = False + got_ids = [(url, repo_identity(url)) for url, _ in id_cases] + print(f" FAIL repo_identity -> {got_ids}") + + # owner_repos: forks are dropped, and pagination stops the moment a page comes back short of 100. + real_gh = globals()["gh"] + try: + pages = { + 1: [{"name": f"r{i}", "fork": False} for i in range(100)], + 2: [{"name": "kept", "fork": False}, {"name": "dropped-fork", "fork": True}], + } + globals()["gh"] = lambda path, ok404=False: ( + {"login": "owner"} if path == "user" else pages[int(path.rsplit("page=", 1)[1])] + ) + got = {r["name"] for r in owner_repos("owner")} + finally: + globals()["gh"] = real_gh + want = {f"r{i}" for i in range(100)} | {"kept"} + if got != want: + ok = False + print( + f" FAIL owner_repos -> {len(got)} repos, expected {len(want)}, forks/pagination wrong" + ) + else: + print(" ok owner_repos: forks dropped, pagination stops on a short page") + + # A gh("user") call that returns None (an empty response body, per its own docstring) must + # raise the clear ownership-mismatch error, not crash with AttributeError on .get(). + try: + globals()["gh"] = lambda path, ok404=False: None if path == "user" else [] + owner_repos("owner") + ok = False + print(" FAIL owner_repos: a None gh('user') response did not raise") + except AttributeError: + ok = False + print(" FAIL owner_repos: a None gh('user') response crashed with AttributeError") + except RuntimeError as exc: + # Any RuntimeError would pass a bare except clause, including an unrelated one this test + # was never meant to exercise. Assert the actual ownership-mismatch text. + if "not registry owner 'owner'" not in str(exc): + ok = False + print(f" FAIL owner_repos: a None gh('user') response raised the wrong error -> {exc}") + else: + print(" ok owner_repos: a None gh('user') response raises the clear mismatch error") + finally: + globals()["gh"] = real_gh + + # GitHub logins are case-insensitive, so a registry owner spelled with different casing than + # gh's own login must still be accepted as the same account. + try: + globals()["gh"] = lambda path, ok404=False: {"login": "Owner"} if path == "user" else [] + owner_repos("owner") + print(" ok owner_repos: a differently-cased login still matches the registry owner") + except RuntimeError as e: + ok = False + print(f" FAIL owner_repos: a differently-cased login was wrongly rejected -> {e}") + finally: + globals()["gh"] = real_gh + + # A None page mid-pagination (an empty response body) must fail loud rather than being read + # as a short page: `or []` there would silently truncate the sweep and report clean. + try: + globals()["gh"] = lambda path, ok404=False: {"login": "owner"} if path == "user" else None + owner_repos("owner") + ok = False + print(" FAIL owner_repos: a None page did not raise") + except RuntimeError: + print(" ok owner_repos: a None page fails loud instead of reading as an empty page") + finally: + globals()["gh"] = real_gh + + real_owner_repos = globals()["owner_repos"] + try: + cases = [ + ( + "a repo missing from the registry is a DEFECT", + [{"name": "New", "full_name": "owner/New", "archived": False}], + {"owner": "owner", "repos": []}, + [("DEFECT", "owner/New: exists on GitHub with no registry/repos.json entry")], + ), + ( + "archived on GitHub but not in the registry is DRIFT", + [{"name": "Old", "full_name": "owner/Old", "archived": True}], + { + "owner": "owner", + "repos": [ + { + "name": "Old", + "url": "https://github.com/owner/Old", + "status": "cataloged", + } + ], + }, + [ + ( + "DRIFT", + "Old: archived on GitHub but registry status is 'cataloged' (reconcile to 'archived')", + ) + ], + ), + ( + "registry says archived but GitHub disagrees is DRIFT", + [{"name": "Back", "full_name": "owner/Back", "archived": False}], + { + "owner": "owner", + "repos": [ + { + "name": "Back", + "url": "https://github.com/owner/Back", + "status": "archived", + } + ], + }, + [ + ( + "DRIFT", + "Back: registry status is 'archived' but GitHub reports it unarchived " + + "(reconcile status, or verify it was not unarchived by mistake)", + ) + ], + ), + ( + "matched, not archived either side, is clean", + [{"name": "Fine", "full_name": "owner/Fine", "archived": False}], + { + "owner": "owner", + "repos": [ + { + "name": "Fine", + "url": "https://github.com/owner/Fine", + "status": "cataloged", + } + ], + }, + [], + ), + ( + "identity matching is case-insensitive", + [{"name": "MixedCase", "full_name": "Owner/MixedCase", "archived": False}], + { + "owner": "owner", + "repos": [ + { + "name": "mixedcase", + "url": "https://github.com/owner/mixedcase", + "status": "cataloged", + } + ], + }, + [], + ), + ( + "an incidental URL whitespace difference does not miss a real match", + [{"name": "Spacey", "full_name": "owner/Spacey", "archived": False}], + { + "owner": "owner", + "repos": [ + { + "name": "Spacey", + "url": " https://github.com/owner/Spacey ", + "status": "cataloged", + } + ], + }, + [], + ), + ( + "a same-named repo under a different owner is not treated as a match", + [{"name": "Shared", "full_name": "owner/Shared", "archived": False}], + { + "owner": "owner", + "repos": [ + { + "name": "Shared", + "url": "https://github.com/someone-else/Shared", + "status": "cataloged", + } + ], + }, + [("DEFECT", "owner/Shared: exists on GitHub with no registry/repos.json entry")], + ), + ] + for label, gh_repos, spec_registry, want in cases: + globals()["owner_repos"] = lambda owner, r=gh_repos: r + got = membership_findings({"registry": spec_registry}) + if got != want: + ok = False + print(f" FAIL membership: {label} -> {got}") + else: + print(f" ok membership: {label}") + finally: + globals()["owner_repos"] = real_owner_repos + print("SELFTEST PASS" if ok else "SELFTEST FAIL") return 0 if ok else 1 @@ -4247,6 +4569,27 @@ def main(argv=None): print() hard = 0 + + # Fleet-wide, not per-repo, so it runs once and only on a full sweep: a name-filtered or + # --issue run is scoped to specific repos already in the registry and has nothing to gain + # from re-listing every repo the owner has. + if not wanted: + try: + membership = membership_findings(spec) + # A gh failure here (auth mismatch, rate limit, network) must not abort the per-repo sweep below. + except Exception as e: # noqa: BLE001 + membership = [("ERROR", str(e))] + print("== Fleet membership (registry/repos.json vs GitHub) ==") + if not membership: + print( + " clean (every non-fork owned repo has a registry entry, and archived status matches GitHub)" + ) + for kind, text in membership: + print(f" {kind:6} {text}") + if kind in ("DEFECT", "LETTER", "ERROR"): + hard += 1 + print() + for entry in repos: model = ( entry.get("workflowModel") diff --git a/spec/validate.py b/spec/validate.py index d720dca4..8f089ecb 100755 --- a/spec/validate.py +++ b/spec/validate.py @@ -21,6 +21,9 @@ WORKFLOW_MODELS = ("release", "operational") RELEASE_TRIGGERS = ("two-phase", "publish-on-merge", "dispatch-only", "none") CONSUMER_MODELS = ("push", "pull") +# Parses owner/repo, lowercased, from a repo's url; a trailing .git is stripped so it still matches GitHub's own full_name. +# A duplicate identity here would let spec/audit.py's fleet membership check silently shadow one entry with the other. +GITHUB_URL_RE = re.compile(r"^https://github\.com/([^/\s]+)/([^/\s]+?)(?:\.git)?/?$") # How faithfully a carried unit is checked, per spec/fidelity-model.md, defaulting to presence. FIDELITIES = ("presence", "intent", "verbatim", "interface") # The keys an interface unit's `contract` may carry (kept in sync with files.schema.json). @@ -423,11 +426,49 @@ def check_secret_set(label, entry, need_kind): f"defaults.releaseTrigger '{default_trigger}' invalid (expected one of {', '.join(RELEASE_TRIGGERS)})" ) + seen_identities = set() for i, repo in enumerate(repos["repos"]): if not isinstance(repo, dict): errors.append(f"repo #{i} is not an object") continue name = repo.get("name", f"#{i}") + # This name only labels every error message below. + # The membership check (spec/audit.py's membership_findings()) keys by owner/repo instead, parsed from url the same way this loop does. + if not isinstance(repo.get("name"), str) or not repo["name"].strip(): + errors.append(f"repo #{i}: missing or empty 'name'") + continue + if not isinstance(repo.get("url"), str) or not repo["url"].strip(): + errors.append(f"{name}: missing or empty 'url'") + continue + m = GITHUB_URL_RE.match(repo["url"].strip()) + if m is None: + # A url that is a well-formed URI but not this exact shape (http://, a path suffix) would otherwise pass here. + # It would only surface later as a false DEFECT, since membership_findings() can never resolve it to an identity. + errors.append(f"{name}: url is not a github.com// URL") + continue + identity = f"{m.group(1)}/{m.group(2)}".lower() + if identity in seen_identities: + errors.append(f"{name}: duplicate registry entry for '{identity}'") + seen_identities.add(identity) + + # These fields are facts about the repo itself, not about its audit scope. + # The schema's operational-needs-lineEndings rule (registry/repos.schema.json) binds regardless of status. + # Checked here, before the status branch, so every status shares one check rather than each non-cataloged branch needing its own copy. + model = repo.get("workflowModel") + if model is not None and model not in WORKFLOW_MODELS: + errors.append( + f"{name}: workflowModel '{model}' invalid (expected {' or '.join(WORKFLOW_MODELS)})" + ) + eol = repo.get("lineEndings") + if eol is not None and eol not in ("lf", "crlf"): + errors.append(f"{name}: lineEndings '{eol}' invalid (expected lf or crlf)") + # An operational repo's endings follow the consuming app's platform, so they must be declared, where a release repo omits the field and takes the fleet LF default. + # Resolve the effective model the way configure.sh does, from the repo, then the defaults, then release. + # The requirement then holds even where a repo relies on an operational defaults.workflowModel rather than setting its own. + effective_model = model or default_model or "release" + if effective_model == "operational" and eol is None: + errors.append(f"{name}: operational repo must declare lineEndings (lf or crlf)") + status = repo.get("status") if status is None: errors.append(f"{name}: missing 'status'") @@ -436,6 +477,15 @@ def check_secret_set(label, entry, need_kind): if not repo.get("classificationPending"): errors.append(f"{name}: backlog repo without classificationPending") continue + if status == "archived": + # GitHub's own archived flag is the fact. + # The entry only needs to exist, so spec/audit.py's fleet membership check has something to match it against. + continue + if status == "excluded": + reason = repo.get("exclusionReason") + if not isinstance(reason, str) or not reason.strip(): + errors.append(f"{name}: excluded repo without a non-empty exclusionReason") + continue if status != "cataloged": errors.append(f"{name}: unknown status '{status}'") continue @@ -463,12 +513,6 @@ def check_secret_set(label, entry, need_kind): f"{name}: type '{tname}' profile '{prof}' not in its allowed profiles {allowed or '[]'}" ) - model = repo.get("workflowModel") - if model is not None and model not in WORKFLOW_MODELS: - errors.append( - f"{name}: workflowModel '{model}' invalid (expected {' or '.join(WORKFLOW_MODELS)})" - ) - # The releaseTrigger field is a scope selector, per spec/scope-model.md, so an invalid value would silently fail to match any releaseTrigger-scoped section rather than error. trigger = repo.get("releaseTrigger") if trigger is not None and trigger not in RELEASE_TRIGGERS: @@ -484,16 +528,6 @@ def check_secret_set(label, entry, need_kind): f"{name}: consumerModel '{cm}' invalid or missing (expected {' or '.join(CONSUMER_MODELS)})" ) - eol = repo.get("lineEndings") - if eol is not None and eol not in ("lf", "crlf"): - errors.append(f"{name}: lineEndings '{eol}' invalid (expected lf or crlf)") - # An operational repo's endings follow the consuming app's platform, so they must be declared, where a release repo omits the field and takes the fleet LF default. - # Resolve the effective model the way configure.sh does, from the repo, then the defaults, then release. - # The requirement then holds even where a repo relies on an operational defaults.workflowModel rather than setting its own. - effective_model = model or default_model or "release" - if effective_model == "operational" and eol is None: - errors.append(f"{name}: operational repo must declare lineEndings (lf or crlf)") - required = set(repo.get("requiredSecrets", [])) for pub in repo.get("publish", []): if not isinstance(pub, dict) or "target" not in pub or "mechanism" not in pub: @@ -857,7 +891,17 @@ def check_selector(where, applies_to): 1 for r in repos["repos"] if isinstance(r, dict) and r.get("status") == "cataloged" ) backlog = sum(1 for r in repos["repos"] if isinstance(r, dict) and r.get("status") == "backlog") - print(f"Spec validation OK: {cataloged} cataloged, {backlog} backlog repos classify cleanly.") + archived = sum( + 1 for r in repos["repos"] if isinstance(r, dict) and r.get("status") == "archived" + ) + excluded = sum( + 1 for r in repos["repos"] if isinstance(r, dict) and r.get("status") == "excluded" + ) + print( + f"Spec validation OK: {cataloged} cataloged repos classify cleanly. " + f"{backlog} backlog repos await classification. " + f"{archived} archived, {excluded} excluded repos carry a valid entry." + ) return 0 From 2e0b908a84f87621831946b4abc940e59ff5c67f Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 22 Aug 2026 08:54:16 -0700 Subject: [PATCH 2/3] Drop a Semicolon From a GITHUB_URL_RE Comment (#911) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A qodo finding on the promotion PR (#910) caught a real semicolon in spec/validate.py's GITHUB_URL_RE comment. scripts/prose_lint.py's dash/semicolon checks are Markdown-only, so it never caught this. Split into two sentences. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Summary by CodeRabbit * **Documentation** * Clarified comments describing GitHub URL normalization, including repository name casing and `.git` suffix handling. --- spec/validate.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/spec/validate.py b/spec/validate.py index 8f089ecb..e838ed2e 100755 --- a/spec/validate.py +++ b/spec/validate.py @@ -21,7 +21,8 @@ WORKFLOW_MODELS = ("release", "operational") RELEASE_TRIGGERS = ("two-phase", "publish-on-merge", "dispatch-only", "none") CONSUMER_MODELS = ("push", "pull") -# Parses owner/repo, lowercased, from a repo's url; a trailing .git is stripped so it still matches GitHub's own full_name. +# Parses owner/repo, lowercased, from a repo's url. +# A trailing .git is stripped so it still matches GitHub's own full_name. # A duplicate identity here would let spec/audit.py's fleet membership check silently shadow one entry with the other. GITHUB_URL_RE = re.compile(r"^https://github\.com/([^/\s]+)/([^/\s]+?)(?:\.git)?/?$") # How faithfully a carried unit is checked, per spec/fidelity-model.md, defaulting to presence. From 39468dccb7eb6c77d76c3b25c85c990bcbf695c9 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 22 Aug 2026 09:34:36 -0700 Subject: [PATCH 3/3] Reject a Query String or Fragment in a Registry URL (#912) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A CodeRabbit finding on the promotion PR (#910) caught a real gap: GITHUB_URL_RE allowed `?` and `#` into the repo-name component, so a url with a query string or fragment produced a wrong identity that could never match GitHub's real full_name, causing a false membership DEFECT. Both regex copies now exclude those characters. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Summary by CodeRabbit * **Bug Fixes** * GitHub repository URLs containing query strings or fragments are now rejected. * Valid HTTPS GitHub URLs with optional trailing `.git` and slash continue to be accepted. * Repository identities no longer incorrectly include query or fragment content. --- spec/audit.py | 14 ++++++++++---- spec/validate.py | 4 +++- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/spec/audit.py b/spec/audit.py index 48e9a387..7006ba2e 100755 --- a/spec/audit.py +++ b/spec/audit.py @@ -255,15 +255,16 @@ def owner_repos(owner): page += 1 -GITHUB_URL_RE = re.compile(r"^https://github\.com/([^/\s]+)/([^/\s]+?)(?:\.git)?/?$") +GITHUB_URL_RE = re.compile(r"^https://github\.com/([^/\s?#]+)/([^/\s?#]+?)(?:\.git)?/?$") def repo_identity(url): """owner/repo, lowercased, parsed from a github.com URL, or None if it does not parse. - The canonical identity two repos are compared by, since a bare repo name collides across - owners and GitHub's own full_name field is already this exact shape (case-preserved). A - trailing .git is stripped so it still resolves to the same identity full_name would. + The canonical identity two repos are compared by, since a bare repo name collides across owners. + GitHub's own full_name field is already this exact shape (case-preserved). + A trailing .git is stripped so it still resolves to the same identity full_name would. + A query string or fragment is rejected outright rather than folded into the repo name. spec/validate.py rejects a url that fails to parse here at all, kept in sync with this regex. """ if not isinstance(url, str): @@ -4212,6 +4213,11 @@ def _selftest(): (" https://github.com/owner/Repo ", "owner/repo"), ("https://github.com/owner/Repo.git", "owner/repo"), ("https://github.com/owner/Repo.github", "owner/repo.github"), + ("https://github.com/owner/Repo?tab=readme", None), + ("https://github.com/owner/Repo#readme", None), + ("https://github.com/owner/Repo.git?tab=readme", None), + ("https://github.com/owner?tab=readme/Repo", None), + ("https://github.com/owner#readme/Repo", None), ("http://github.com/owner/Repo", None), ("https://gitlab.com/owner/Repo", None), ("not a url", None), diff --git a/spec/validate.py b/spec/validate.py index e838ed2e..446f144e 100755 --- a/spec/validate.py +++ b/spec/validate.py @@ -23,8 +23,10 @@ CONSUMER_MODELS = ("push", "pull") # Parses owner/repo, lowercased, from a repo's url. # A trailing .git is stripped so it still matches GitHub's own full_name. +# A query character or a fragment character is excluded from both groups too. +# Otherwise a query string or fragment folds into the repo name instead of failing to match. # A duplicate identity here would let spec/audit.py's fleet membership check silently shadow one entry with the other. -GITHUB_URL_RE = re.compile(r"^https://github\.com/([^/\s]+)/([^/\s]+?)(?:\.git)?/?$") +GITHUB_URL_RE = re.compile(r"^https://github\.com/([^/\s?#]+)/([^/\s?#]+?)(?:\.git)?/?$") # How faithfully a carried unit is checked, per spec/fidelity-model.md, defaulting to presence. FIDELITIES = ("presence", "intent", "verbatim", "interface") # The keys an interface unit's `contract` may carry (kept in sync with files.schema.json).