Skip to content

Add fidelity-honesty analysis tool (owner-run) - #387

Merged
ptr727 merged 8 commits into
developfrom
fidelity-honesty-analysis
Jul 22, 2026
Merged

Add fidelity-honesty analysis tool (owner-run)#387
ptr727 merged 8 commits into
developfrom
fidelity-honesty-analysis

Conversation

@ptr727

@ptr727 ptr727 commented Jul 22, 2026

Copy link
Copy Markdown
Owner

A read-only fleet analysis that checks the manifest's declared fidelities against reality — the declared→verified leap one level up from the verbatim engine (which verifies content). Reuses spec/audit.py's machinery. Owner-run, not wired into CI.

What it reports

  • intent units byte-uniform fleet-wide → candidates to promote to verbatim (gaining free drift-detection).
  • verbatim units with non-stale downstream divergence → mis-label or real drift.
  • Manifest gaps — files carried by a reference adopter but absent from the manifest (the configure.sh bug class).

First-run value (findings, not fixed here)

  • 3 ruleset payloads (repo-config/{develop,operational/develop,main}.json) are uniform → verbatim candidates.
  • .markdownlint-cli2.jsonc and configure.sh carry real downstream drift (a re-vendor worklist).
  • .editorconfig-checker.json is carried but untracked — a new manifest gap.

Follow-ups (each its own decision) are tracked separately. This PR just lands the tool.

🤖 Generated with Claude Code

Read-only fleet analysis that checks the manifest's declared fidelities against
reality - the declared-to-verified leap one level up from the verbatim engine
(which verifies content). Reuses spec/audit.py's machinery.

For each intent/verbatim unit it fetches every applicable cataloged repo's copy
and reports: intent units that are byte-uniform fleet-wide (candidates to promote
to verbatim, gaining free drift-detection), verbatim units with non-stale
downstream divergence (mis-label or drift), and a manifest-gap pass (files
carried by a reference adopter but absent from the manifest - the configure.sh
bug class). Not wired into CI; prototype for review.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 13:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an owner-run, read-only fleet analysis script that reuses spec/audit.py helpers to compare spec/files.json declared fidelities (intent/verbatim) against observed downstream content hashes, and to detect manifest gaps via a reference adopter.

Changes:

  • Add spec/fidelity_honesty.py to compute per-unit fleet spread, promotion candidates, and potential mislabels.
  • Add a manifest-gap pass that compares hub files against a reference adopter's repo tree.
  • Print a human-readable report summarizing the findings.

Comment thread spec/fidelity_honesty.py
Comment thread spec/fidelity_honesty.py Outdated
Comment thread spec/fidelity_honesty.py Outdated
- manifest_gap_pass resolved the git/trees endpoint with a branch name; that can
  404 and silently drop the check. Resolve the branch to its tree SHA first, as
  audit.py does (also removes a possibly-unbound local).
- Pin spec/fidelity_honesty.py to LF in .editorconfig alongside its audit.py /
  validate.py siblings, so its shebang stays executable.
- Reword "byte-uniform" -> "content-identical (after EOL normalization)": the
  comparison is content_hash, which normalizes line endings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 13:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Comment thread spec/fidelity_honesty.py Outdated
Comment thread spec/fidelity_honesty.py Outdated
Comment thread .editorconfig
…ributes (#387)

- The manifest-gap pass excluded paths with `.startswith(".git")`, which also
  swallowed `.github/`, `.gitattributes`, and `.gitignore` - hiding real gaps.
  And the hub-side `.git/` substring exclusion was path-separator-dependent.
  Exclude the .git metadata dir by path component (`".git" not in p.parts`) and
  drop the startswith filter. This immediately surfaced two previously-hidden
  gaps: .github/workflows/publish-release.yml and validate-task.yml are carried
  but untracked.
- Add spec/fidelity_honesty.py to the .gitattributes LF-pin, matching its
  audit.py / validate.py / install.py siblings (git-enforced, not just editor).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 14:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread spec/fidelity_honesty.py Outdated
Comment thread spec/fidelity_honesty.py
Comment thread spec/fidelity_honesty.py
…le" bucket (#387)

- The verbatim-candidate detector treated any present copy as evidence of
  uniformity, so a unit that was entirely stale/unavailable could be flagged.
  Require at least one confirmed match with the current canonical (still allowing
  stale copies, which verbatim would flag "stale -> re-vendor"), and reword the
  header to stop implying "fleet-wide uniform".
- Rename the "absent" bucket to "unavailable": fetch() returns None for both a
  missing file and a present-but-non-inline response (too large / encoding
  "none"), so "absent" over-claimed. The report now labels it "absent or
  non-inline".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 14:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

spec/fidelity_honesty.py:55

  • The inline comment for promote still says these intent units are "uniform fleet-wide", but the actual criteria only requires >=1 confirmed match with the current canonical and 0 "differs" (stale/unavailable are allowed). This makes the comment misleading for future readers.
    spreads = []          # (entry, spread dict) for the full table
    promote = []          # intent units that are uniform fleet-wide
    mislabel = []         # verbatim units that diverge non-stale

Comment thread spec/fidelity_honesty.py
Comment thread spec/fidelity_honesty.py Outdated
…lk (#387)

manifest_gap_pass built the hub-file set with ROOT.rglob('*'), a filesystem walk
that pulls in untracked local cruft (__pycache__ from running the tool, a local
.venv) and makes the gap report depend on working-tree state. Use git ls-files -
the tracked set, which is what "carried" actually means and is deterministic
regardless of local files. Same gaps result, without the walk's noise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 14:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

spec/fidelity_honesty.py:106

  • manifest_gap_pass() returns an empty gaps list when the branch/tree lookup fails (404 / missing fields). main() then prints "none - the manifest covers...", which is a false negative because the check did not actually run. Also, the git/trees API can return {"truncated": true}; in that case the gap set would be incomplete and should fail loud rather than reporting "none".
    br = audit.gh(f"repos/{slug}/branches/{ground}", ok404=True)
    if not br or "commit" not in br:
        return slug, []
    tree = audit.gh(f"repos/{slug}/git/trees/{br['commit']['commit']['tree']['sha']}?recursive=1", ok404=True)
    if not tree or "tree" not in tree:
        return slug, []

spec/fidelity_honesty.py:54

  • The inline comment for promote says "uniform fleet-wide", but the logic intentionally allows stale/unavailable repos and only requires >=1 canonical match and 0 hand-modified copies. Updating the comment avoids misreading what promote represents.
    promote = []          # intent units that are uniform fleet-wide

Comment thread spec/fidelity_honesty.py Outdated
An empty hub-file set would make the manifest-gap report show "none" - a false
clean. Raise on a non-zero git ls-files exit instead, matching audit.py's gh().

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 14:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

spec/fidelity_honesty.py:54

  • The inline comment says intent units in promote are "uniform fleet-wide", but the promote condition allows stale/unavailable copies (it only requires >=1 match and 0 confirmed divergent copies). This comment is misleading and contradicts the logic below.
    promote = []          # intent units that are uniform fleet-wide

Comment thread spec/fidelity_honesty.py Outdated
…#387)

If the reference adopter's branch or tree lookup returned nothing,
manifest_gap_pass returned an empty gaps list, which main() reports as "none" -
a false clean. Raise instead, matching the git ls-files guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 14:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

spec/fidelity_honesty.py:55

  • The promote list no longer implies fleet-wide uniformity (units can be unavailable in some repos), but the comment still says "uniform fleet-wide", which is misleading.
    promote = []          # intent units that are uniform fleet-wide
    mislabel = []         # verbatim units that diverge non-stale

Comment thread spec/fidelity_honesty.py Outdated
Comment thread spec/fidelity_honesty.py Outdated
Comment thread spec/fidelity_honesty.py Outdated
…ons (#387)

Three semicolons in this file's comments and output string joined independent
clauses, which the AGENTS.md prose rule bans. Recast to commas/periods.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 14:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@ptr727
ptr727 merged commit 825afea into develop Jul 22, 2026
7 checks passed
@ptr727
ptr727 deleted the fidelity-honesty-analysis branch July 22, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants