Add the verbatim engine - content-hash fixed units against canonical (#374, PR-C)

AUDIT.md

@@ -10,7 +10,7 @@ The verdict vocabulary is [`WORKFLOW.md`][workflow]'s: **operational / not opera
 This audit is not occasional. Run it whenever you **create, adopt, or materially change** a fleet repo, and on demand for any known repo:
 
 - **Onboarding a repo is complete only when it either passes this audit** (operational - every applicable check) **or carries a committed `reports/<repo>/audit.md` plus a tracking issue** enumerating every residual delta. A repo that is partially set up but never audited is itself a **defect** - the exact state this process prevents. The create-to-conformance counterpart is [`STANDUP.md`][standup]; because both read the same manifests, a repo stood up by that file passes this audit by construction.
-- **Touching a repo** (any conformance-affecting change) ends by re-running the applicable checks and **reconciling the registry entry to reality** - `status`, `types`, `releaseTrigger`, `workflowModel`, `driftNotes`. The registry records reality, not intent. [`spec/validate.py`][validate] proves the catalog is self-consistent, not that it matches the live repo - closing that gap is this audit's job. The deterministic subset (settings, rulesets, secret names, file presence, per-scope markdown section presence, workflow interface conformance, branch facts) is mechanized in [`spec/audit.py`][audit-runner]: owner-initiated, run on demand when onboarding a repo, on suspected drift, or before fleet-wide changes. A required section missing from a carried markdown file is a **drift finding**, not a letter - a heading rename reads as missing, and equivalence is judged by hand. A carried `interface` workflow (spec/fidelity-model.md) is checked by name and wiring - required jobs, the ruleset-bound check name, the artifact-name handoff, and the forbidden `artifact-ids:` fork - all at **drift**, since the body is owned and a rename is a hint to verify.
+- **Touching a repo** (any conformance-affecting change) ends by re-running the applicable checks and **reconciling the registry entry to reality** - `status`, `types`, `releaseTrigger`, `workflowModel`, `driftNotes`. The registry records reality, not intent. [`spec/validate.py`][validate] proves the catalog is self-consistent, not that it matches the live repo - closing that gap is this audit's job. The deterministic subset (settings, rulesets, secret names, file presence, per-scope markdown section presence, workflow interface conformance, verbatim content, branch facts) is mechanized in [`spec/audit.py`][audit-runner]: owner-initiated, run on demand when onboarding a repo, on suspected drift, or before fleet-wide changes. A required section missing from a carried markdown file is a **drift finding**, not a letter - a heading rename reads as missing, and equivalence is judged by hand. A carried `interface` workflow (spec/fidelity-model.md) is checked by name and wiring - required jobs, the ruleset-bound check name, the artifact-name handoff, and the forbidden `artifact-ids:` fork - all at **drift**, since the body is owned and a rename is a hint to verify. A carried `verbatim` unit - a whole file (`.markdownlint-cli2.jsonc`) or a canonical workflow job region (the `github-release` job) - is content-hashed against the hub's canonical after line-ending normalization. A mismatch is classified **stale** (matches a past hub revision, re-vendor) or **modified** (matches none, the repo changed fixed content), both at **drift**, since equivalence is intent-governed and a byte diff is a hint to review.
 
 ## 1. Scope and Ground-Truth Branch
 

spec/audit.py

@@ -18,6 +18,8 @@
 Usage: python3 spec/audit.py [RepoName ...]   (default: every cataloged repo)
 """
 import base64
+import functools
+import hashlib
 import json
 import pathlib
 import re
@@ -242,6 +244,87 @@ def check_interface(path, contract, text):
     return findings
 
 
+def normalize(text):
+    """Reduce a carried unit to its comparable form: neutralize line endings, since EOL variance is governed
+    separately, not a fidelity deviation. No placeholder masking - see spec/fidelity-model.md "Normalization".
+    """
+    return text.replace("\r\n", "\n").replace("\r", "\n")
+
+
+@functools.lru_cache(maxsize=1024)  # bounded; the keys that recur across repos are the canonical and its history
+def _hash_normalized(norm_text):
+    return hashlib.sha256(norm_text.encode("utf-8")).hexdigest()
+
+
+def content_hash(text):
+    # Cache on the normalized form, not raw text, so EOL-only variants (CRLF vs LF) share one entry.
+    return _hash_normalized(normalize(text))
+
+
+def classify_verbatim(down_text, canon_text, past_texts):
+    """None if the downstream copy matches the current canonical, 'stale' if it matches a past hub revision
+    (the base advanced - re-vendor), or 'modified' if it matches no revision the base ever produced (the
+    repo changed fixed content). The discriminator is a content hash, never a version stamp - a stamp can
+    claim to be current while the body was edited, so it is never trusted for integrity.
+    """
+    dh = content_hash(down_text)
+    if dh == content_hash(canon_text):
+        return None
+    for past in past_texts:
+        if content_hash(past) == dh:
+            return "stale"
+    return "modified"
+
+
+_HISTORY_CACHE = {}  # rel_path -> [past revision content], reused as a canonical is compared against every audited repo
+
+
+def git_file_history(rel_path):
+    """Every past revision's content of a hub-tracked file (to tell a stale copy from a modified one), cached per rel_path."""
+    if rel_path in _HISTORY_CACHE:
+        return _HISTORY_CACHE[rel_path]
+    out = []
+    # Decode as UTF-8/replace to match the downstream and canonical reads. A divergent decode would fabricate a mismatch.
+    r = subprocess.run(["git", "log", "--format=%H", "--", rel_path], cwd=ROOT, capture_output=True,
+                       encoding="utf-8", errors="replace")
+    if r.returncode == 0:
+        for sha in r.stdout.split():
+            s = subprocess.run(["git", "show", f"{sha}:{rel_path}"], cwd=ROOT, capture_output=True,
+                               encoding="utf-8", errors="replace")
+            if s.returncode == 0:
+                out.append(s.stdout)
+    _HISTORY_CACHE[rel_path] = out
+    return out
+
+
+def check_verbatim(label, down_text, canonical_rel, extract=None):
+    """Compare a downstream copy against the hub's canonical (a region if `extract` is given), EOL-normalized,
+    and classify a mismatch as stale or modified via the canonical's git history. All findings are DRIFT: a
+    byte diff is a hint to review, never proof of breakage.
+    """
+    try:
+        # Same decode policy as the downstream copy and the git history, so a stray byte can never make
+        # otherwise-equal content hash differently across the three sources.
+        canon_text = (ROOT / canonical_rel).read_text(encoding="utf-8", errors="replace")
+    except OSError:
+        return [("DRIFT", f"verbatim: {label} canonical {canonical_rel} is unreadable from the hub (spec error?)")]
+    history = git_file_history(canonical_rel)
+    if extract is not None:
+        down_region, canon_region = extract(down_text), extract(canon_text)
+        if canon_region is None:
+            return [("DRIFT", f"verbatim: {label} region absent in the canonical (spec error?)")]
+        if down_region is None:
+            return [("DRIFT", f"verbatim: {label} region absent downstream, cannot compare")]
+        down_text, canon_text = down_region, canon_region
+        history = [h for h in (extract(t) for t in history) if h is not None]
+    verdict = classify_verbatim(down_text, canon_text, history)
+    if verdict is None:
+        return []
+    if verdict == "stale":
+        return [("DRIFT", f"verbatim: {label} matches a past hub revision, not the current canonical - the base advanced, re-vendor it")]
+    return [("DRIFT", f"verbatim: {label} differs from the canonical and matches no past hub revision - the repo modified fixed content, review it")]
+
+
 def audit_repo(entry, spec):
     findings = []  # (kind, text)
     slug = repo_slug(entry)
@@ -395,7 +478,7 @@ def audit_repo(entry, spec):
     # is DRIFT (a hint to verify), never a LETTER.
     sel = repo_selectors(entry, spec["registry"].get("defaults", {}))
     wanted_sections = {}  # path -> set of required section names, unioned across applicable entries
-    interface_contract = {}  # path -> contract, for fidelity:interface entries
+    check_item = {}  # path -> entry, for a fidelity interface/verbatim entry (last applicable wins per path)
     path_order = []
     for item in spec["files"]["baseline"]:
         if not applies(item.get("appliesTo", "*"), sel):
@@ -405,37 +488,51 @@ def audit_repo(entry, spec):
             wanted_sections[path] = set()
             path_order.append(path)
         wanted_sections[path].update(required_sections(item, sel))
-        if item.get("fidelity") == "interface":
-            interface_contract[path] = item.get("contract", {})
+        if item.get("fidelity") in ("interface", "verbatim"):
+            check_item[path] = item
     for path in path_order:
         content = gh(f"repos/{slug}/contents/{path}?ref={ground}", ok404=True)
+        item = check_item.get(path)
+        fid = item.get("fidelity") if item else "presence"
         if content is None:
             # An interface unit's presence is DRIFT, not LETTER - a workflow's naming is more variable than a
-            # carried config, so treat absence as a hint to verify rather than a hard defect.
-            if path in interface_contract:
+            # carried config, so absence is a hint to verify. Any other unit's absence is a file-presence LETTER.
+            if fid == "interface":
                 findings.append(("DRIFT", f"interface: {path} absent on {ground}, cannot verify its contract"))
             else:
                 findings.append(("LETTER", f"file: {path} absent on {ground} (verify intent per AUDIT.md section 7)"))
             continue
-        # Interface conformance: check the fixed contract by name and wiring, never the body.
-        if path in interface_contract:
-            body = content.get("content")
-            if not body:
+        # Guard on encoding, not truthiness: an empty file returns encoding "base64" with content "" (decode it
+        # to ""), whereas a too-large or non-inline payload returns encoding "none" (text stays None -> flagged).
+        text = base64.b64decode(content["content"]).decode("utf-8", "replace") if content.get("encoding") == "base64" else None
+        # Interface conformance (name + wiring) plus any verbatim job regions the contract pins.
+        if fid == "interface":
+            if text is None:
                 findings.append(("DRIFT", f"interface: could not read {path} content on {ground} to verify its contract (no inline content returned); verify by hand"))
             else:
-                findings.extend(check_interface(path, interface_contract[path], base64.b64decode(body).decode("utf-8", "replace")))
+                contract = item.get("contract", {})
+                findings.extend(check_interface(path, contract, text))
+                canonical_rel = item.get("reference") or path
+                for job in contract.get("verbatimJobs", []):
+                    findings.extend(check_verbatim(f"{path} job '{job}'", text, canonical_rel,
+                                                   extract=lambda t, j=job: split_jobs(t).get(j)))
+        # Whole-file verbatim: byte-identical to the hub's canonical after EOL normalization.
+        elif fid == "verbatim":
+            if text is None:
+                findings.append(("DRIFT", f"verbatim: could not read {path} content on {ground} to compare (no inline content returned); verify by hand"))
+            else:
+                findings.extend(check_verbatim(path, text, item.get("reference") or path))
         # Heading-based presence is only meaningful for markdown. A "section" named on a non-md file (e.g. a
         # tasks.json task group) is an intent marker judged per AUDIT.md, not a heading grep.
         needed = wanted_sections[path]
         if needed and path.endswith(".md"):
-            body = content.get("content")
-            if not body:
+            if text is None:
                 # Fail loud rather than skip silently: the contents API returned no inline content (an
                 # oversized file, a symlink, a submodule), so the section check could not run - surface that
                 # instead of a false clean.
                 findings.append(("DRIFT", f"section: could not read {path} content on {ground} to verify sections (no inline content returned); verify by hand"))
             else:
-                present = heading_texts(base64.b64decode(body).decode("utf-8", "replace"))
+                present = heading_texts(text)
                 for name in sorted(needed):
                     if name.strip().lower() not in present:
                         findings.append(("DRIFT", f"section: '{name}' not found as a heading in {path} on {ground} (renamed or missing; verify intent per AUDIT.md section 7)"))
@@ -502,6 +599,35 @@ def _selftest():
         print(f"  FAIL split_jobs (inline mapping) -> {sorted(inline)}")
     else:
         print("  ok   split_jobs (inline-mapping job captured with its content)")
+
+    # Verbatim engine: EOL normalization, hashing, and the stale-vs-modified classification. Exercised here
+    # rather than only in production, because a latent bug in the comparison would otherwise surface as a
+    # false clean on a real fleet run.
+    canon = "line one\nline two\nline three\n"
+    verbatim_cases = [
+        # (label, down_text, canon_text, history, want)
+        ("identical -> match", canon, canon, [], None),
+        ("EOL-only diff (CRLF) -> match", canon.replace("\n", "\r\n"), canon, [], None),
+        ("EOL-only diff (bare CR) -> match", canon.replace("\n", "\r"), canon, [], None),
+        ("body edit -> modified", canon.replace("line two", "line TWO edited"), canon, [], "modified"),
+        ("matches a past revision -> stale", "old body\n", "current body\n", ["old body\n", "older\n"], "stale"),
+        ("matches a past revision modulo EOL -> stale", "old body\r\n", "current body\n", ["old body\n"], "stale"),
+        ("edit in no revision -> modified", "never existed\n", "current body\n", ["old body\n"], "modified"),
+    ]
+    for label, down, canon_t, history, want in verbatim_cases:
+        got = classify_verbatim(down, canon_t, history)
+        if got != want:
+            ok = False
+        print(f"  {'ok  ' if got == want else 'FAIL'} want={str(want):>8} got={str(got):>8}  verbatim: {label}")
+    # Region extraction and hashing: a forked github-release block must hash differently from the canonical.
+    region = split_jobs(rel_ok).get("github-release")
+    forked_region = split_jobs(rel_ok.replace("          merge-multiple: true\n", "          artifact-ids: 1\n")).get("github-release")
+    if region is None or forked_region is None or content_hash(region) == content_hash(forked_region):
+        ok = False
+        print("  FAIL verbatim: forked github-release region should hash differently")
+    else:
+        print("  ok   verbatim: a forked github-release region hashes differently from the canonical")
+
     print("SELFTEST PASS" if ok else "SELFTEST FAIL")
     return 0 if ok else 1
 

spec/fidelity-model.md

@@ -12,7 +12,7 @@ Each [`spec/files.json`][files] entry declares one `fidelity`, defaulting to `pr
 
 - **presence** - the unit exists (a file, or a markdown section heading). The audit's baseline check.
 - **intent** - carried faithfully but judged by meaning, not bytes. A downstream copy legitimately differs (a governed divergence or a paraphrase), and equivalence is a human call via `intentRef`. The audit asserts nothing beyond presence.
-- **verbatim** - byte-identical to the hub's canonical after declared-placeholder normalization. The audit content-hashes the downstream copy against canonical. It applies to a whole file or a stable-handle region (a markdown section by heading, a workflow job by key).
+- **verbatim** - byte-identical to the hub's canonical after line-ending normalization. The audit content-hashes the downstream copy against canonical. It applies to a whole file or a workflow job region (a job selected by key).
 - **interface** - an overridable body that must honor a named contract. The audit checks the contract by name and wiring, never the body.
 
 Fidelity is a declared field defaulting to `presence`, never inferred from `whole`/`placeholders`. `.editorconfig` and `.markdownlint-cli2.jsonc` are both whole with no placeholders yet sit at opposite fidelity, because the discriminator is governance, not field shape.
@@ -28,11 +28,11 @@ Fidelity is a declared field defaulting to `presence`, never inferred from `whol
 
 The fixed interface of a workflow is stated in [`AGENTS.md`][agents] ("Orchestration vs. build - the override seam" and "Workflow YAML Conventions"), and the `interface` check enforces it by name and structure: the ruleset-bound required check `name: Check pull request workflow status job`, the `github-release` and `get-version` job keys, the `release-asset-<branch>-<target>` artifact-name handoff, and that `github-release` collects assets by `pattern:` / `merge-multiple:` and never by an `artifact-ids:` that names a build job's output. A repo owns the leaf `build-<target>-task` job list, its `needs` targets, and its paths-filter, and none of those are checked.
 
-## Placeholder Normalization
+## Normalization
 
-A verbatim check normalizes only the tokens a unit **declares** in its `placeholders` list, never a blanket `<...>` regex. The declared tokens are literal strings (for example `<owner>`, `<repo>`, `<N>`), so masking touches exactly those and leaves intact the sibling metavariables a doc uses in prose (for example `<PATH>`, `<SHA>`). Line endings are neutralized before hashing, because EOL variance is governed by the line-ending rules, not a fidelity deviation.
+A verbatim check compares content by hash after **line-ending normalization only** - EOL variance is governed by the line-ending rules, not a fidelity deviation. It does **not** mask placeholders: a verbatim unit carries none. The files that declare a `placeholders` list (for example `.github/copilot-instructions.md` with `<owner>`, `<repo>`, `<N>`) are fidelity `intent`, judged by hand and never hashed. Masking could not serve a hash anyway - a downstream copy holds the substituted value (`ptr727`), not the token (`<owner>`), so masking the token in the canonical alone would guarantee a mismatch. A verbatim unit that ever needed a per-repo substitution would require template-matching (the canonical as a pattern, the copy as an instance), not this content hash. None does today.
 
-## Stale Versus Violated
+## Stale Versus Modified
 
 A verbatim mismatch is one of two things, told apart **by hash, not by a version**. The audit hashes each past revision of the hub's canonical from its own git history. If the downstream copy matches a **past** canonical revision, the base advanced and the copy is **stale** - re-vendor it. If it matches **no** revision the base ever produced, the repo **modified fixed content** - review it. A version stamp could claim to be current while being neither, so it is demoted to a human-facing label and never consulted for integrity.