Add fidelity-honesty analysis tool (owner-run) - #387
Conversation
Read-only fleet analysis that checks the manifest's declared fidelities against reality - the declared-to-verified leap one level up from the verbatim engine (which verifies content). Reuses spec/audit.py's machinery. For each intent/verbatim unit it fetches every applicable cataloged repo's copy and reports: intent units that are byte-uniform fleet-wide (candidates to promote to verbatim, gaining free drift-detection), verbatim units with non-stale downstream divergence (mis-label or drift), and a manifest-gap pass (files carried by a reference adopter but absent from the manifest - the configure.sh bug class). Not wired into CI; prototype for review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds an owner-run, read-only fleet analysis script that reuses spec/audit.py helpers to compare spec/files.json declared fidelities (intent/verbatim) against observed downstream content hashes, and to detect manifest gaps via a reference adopter.
Changes:
- Add
spec/fidelity_honesty.pyto compute per-unit fleet spread, promotion candidates, and potential mislabels. - Add a manifest-gap pass that compares hub files against a reference adopter's repo tree.
- Print a human-readable report summarizing the findings.
- manifest_gap_pass resolved the git/trees endpoint with a branch name; that can 404 and silently drop the check. Resolve the branch to its tree SHA first, as audit.py does (also removes a possibly-unbound local). - Pin spec/fidelity_honesty.py to LF in .editorconfig alongside its audit.py / validate.py siblings, so its shebang stays executable. - Reword "byte-uniform" -> "content-identical (after EOL normalization)": the comparison is content_hash, which normalizes line endings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ributes (#387) - The manifest-gap pass excluded paths with `.startswith(".git")`, which also swallowed `.github/`, `.gitattributes`, and `.gitignore` - hiding real gaps. And the hub-side `.git/` substring exclusion was path-separator-dependent. Exclude the .git metadata dir by path component (`".git" not in p.parts`) and drop the startswith filter. This immediately surfaced two previously-hidden gaps: .github/workflows/publish-release.yml and validate-task.yml are carried but untracked. - Add spec/fidelity_honesty.py to the .gitattributes LF-pin, matching its audit.py / validate.py / install.py siblings (git-enforced, not just editor). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…le" bucket (#387) - The verbatim-candidate detector treated any present copy as evidence of uniformity, so a unit that was entirely stale/unavailable could be flagged. Require at least one confirmed match with the current canonical (still allowing stale copies, which verbatim would flag "stale -> re-vendor"), and reword the header to stop implying "fleet-wide uniform". - Rename the "absent" bucket to "unavailable": fetch() returns None for both a missing file and a present-but-non-inline response (too large / encoding "none"), so "absent" over-claimed. The report now labels it "absent or non-inline". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
spec/fidelity_honesty.py:55
- The inline comment for
promotestill says these intent units are "uniform fleet-wide", but the actual criteria only requires >=1 confirmed match with the current canonical and 0 "differs" (stale/unavailable are allowed). This makes the comment misleading for future readers.
spreads = [] # (entry, spread dict) for the full table
promote = [] # intent units that are uniform fleet-wide
mislabel = [] # verbatim units that diverge non-stale
…lk (#387) manifest_gap_pass built the hub-file set with ROOT.rglob('*'), a filesystem walk that pulls in untracked local cruft (__pycache__ from running the tool, a local .venv) and makes the gap report depend on working-tree state. Use git ls-files - the tracked set, which is what "carried" actually means and is deterministic regardless of local files. Same gaps result, without the walk's noise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
spec/fidelity_honesty.py:106
- manifest_gap_pass() returns an empty gaps list when the branch/tree lookup fails (404 / missing fields). main() then prints "none - the manifest covers...", which is a false negative because the check did not actually run. Also, the git/trees API can return {"truncated": true}; in that case the gap set would be incomplete and should fail loud rather than reporting "none".
br = audit.gh(f"repos/{slug}/branches/{ground}", ok404=True)
if not br or "commit" not in br:
return slug, []
tree = audit.gh(f"repos/{slug}/git/trees/{br['commit']['commit']['tree']['sha']}?recursive=1", ok404=True)
if not tree or "tree" not in tree:
return slug, []
spec/fidelity_honesty.py:54
- The inline comment for
promotesays "uniform fleet-wide", but the logic intentionally allows stale/unavailable repos and only requires >=1 canonical match and 0 hand-modified copies. Updating the comment avoids misreading whatpromoterepresents.
promote = [] # intent units that are uniform fleet-wide
An empty hub-file set would make the manifest-gap report show "none" - a false clean. Raise on a non-zero git ls-files exit instead, matching audit.py's gh(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
spec/fidelity_honesty.py:54
- The inline comment says intent units in
promoteare "uniform fleet-wide", but the promote condition allows stale/unavailable copies (it only requires >=1 match and 0 confirmed divergent copies). This comment is misleading and contradicts the logic below.
promote = [] # intent units that are uniform fleet-wide
…#387) If the reference adopter's branch or tree lookup returned nothing, manifest_gap_pass returned an empty gaps list, which main() reports as "none" - a false clean. Raise instead, matching the git ls-files guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
spec/fidelity_honesty.py:55
- The
promotelist no longer implies fleet-wide uniformity (units can be unavailable in some repos), but the comment still says "uniform fleet-wide", which is misleading.
promote = [] # intent units that are uniform fleet-wide
mislabel = [] # verbatim units that diverge non-stale
…ons (#387) Three semicolons in this file's comments and output string joined independent clauses, which the AGENTS.md prose rule bans. Recast to commas/periods. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A read-only fleet analysis that checks the manifest's declared fidelities against reality — the declared→verified leap one level up from the verbatim engine (which verifies content). Reuses
spec/audit.py's machinery. Owner-run, not wired into CI.What it reports
intentunits byte-uniform fleet-wide → candidates to promote toverbatim(gaining free drift-detection).verbatimunits with non-stale downstream divergence → mis-label or real drift.configure.shbug class).First-run value (findings, not fixed here)
repo-config/{develop,operational/develop,main}.json) are uniform → verbatim candidates..markdownlint-cli2.jsoncandconfigure.shcarry real downstream drift (a re-vendor worklist)..editorconfig-checker.jsonis carried but untracked — a new manifest gap.Follow-ups (each its own decision) are tracked separately. This PR just lands the tool.
🤖 Generated with Claude Code