Promote develop to main: fidelity tooling, manifest hardening, divergence ledger

.editorconfig

@@ -60,7 +60,7 @@ end_of_line = lf
 # Python is CRLF by the `[*]` default (universal newlines; commonly edited on Windows). Pin LF
 # only for a `.py` executed directly via its shebang, by path - here the CI validation entry point
 # and the fleet-audit runner.
-[spec/{validate,audit}.py]
+[spec/{validate,audit,fidelity_honesty}.py]
 end_of_line = lf
 
 # The agent-safety kit's Python is shebang-executable tooling run by path (the PreToolUse hook and its

.gitattributes

@@ -19,6 +19,7 @@ catalog/snippets/husky/pre-commit text eol=lf
 # installer. Do not re-add a blanket `*.py text eol=lf`.
 spec/validate.py text eol=lf
 spec/audit.py text eol=lf
+spec/fidelity_honesty.py text eol=lf
 host-setup/agent-safety/gh-write-guard.py text eol=lf
 host-setup/agent-safety/install.py text eol=lf
 

CODESTYLE.md

@@ -355,7 +355,7 @@ This is the style guide for any **Python project(s)** in this repo.
 **Two profiles.** A repo's Python is one of two shapes, and the rest of this section (uv project, `uv.lock`, `uv run`, `src` layout, pytest coverage) describes the **project** profile. The two differ by whether the Python has **third-party runtime dependencies**, which shows up structurally in `pyproject.toml`, so the audit detects the profile there (`python.profile.detect`):
 
 - **Project** - the Python has third-party runtime dependencies, or is the repo's deliverable. It is a PEP 621 uv project: `[project]` with `dependencies` (dev tools in `[project.optional-dependencies]` or `[dependency-groups]`), a `[build-system]`, and a committed `uv.lock` (pinned LF - see [Line Endings][line-endings]); CI runs `uv sync --frozen` + `uv run <tool>`, so the lockfile pins tool versions.
-- **Scripts** - stdlib-only utility scripts embedded in a **non-Python** repo (e.g. a Python tooling subtree of a `csharp` app). Run the tools with **`uvx`** (no project install, no lockfile): the `pyproject.toml` carries **only** `[tool.ruff]` / `[tool.mypy]` config - no `[project]`, no `[build-system]`, no `uv.lock` (that metadata would misrepresent it as a shippable package). **mypy** is the type checker (there is no first-party package for pyright strict to anchor on). Because there is no lockfile to pin versions, **CI pins the exact tool versions in the `uvx` command** (`uvx ruff@<ver>`, `uvx mypy@<ver>`, bumpable there) while the VS Code tasks and README run the unpinned latest - a deliberate CI-vs-local gap so local tooling never silently falls behind. `.py` files follow the repo's line-ending default (CRLF in a CRLF-default repo; a shebang-executed script is LF-pinned by path - see [Line Endings][line-endings]). There is no pytest suite, so the coverage expectation is N/A; a co-present `csharp` type still carries `codecov.yml` for its own tests.
+- **Scripts** - stdlib-only utility scripts embedded in a **non-Python** repo (e.g. a Python tooling subtree of a `csharp` app). Run the tools with **`uvx`** (no project install, no lockfile): the `pyproject.toml` carries **only** tool config (`[tool.ruff]`, `[tool.mypy]`, and an optional `[tool.pyright]` editor block) - no `[project]`, no `[build-system]`, no `uv.lock` (that metadata would misrepresent it as a shippable package). **mypy** is the type-check gate (there is no first-party package for pyright strict to anchor on), and a `[tool.pyright]` block in **standard** mode keeps Pylance quiet in the editor - the same mypy-gate/pyright-editor split the Project profile uses. There is no lockfile, and a `uvx <tool>@<ver>` pin in a `run:` step is not something Dependabot tracks, so **CI runs `uvx ruff@latest` / `uvx mypy@latest`** rather than a manual pin that would silently go stale. The fleet rule is to pin only what Dependabot auto-updates (SHA-pinned actions, package deps) and otherwise run latest, so the VS Code tasks, README, and CI all run the unpinned latest here. `.py` files follow the repo's line-ending default (CRLF in a CRLF-default repo, and a shebang-executed script is LF-pinned by path - see [Line Endings][line-endings]). There is no pytest suite, so the coverage expectation is N/A. A co-present `csharp` type still carries `codecov.yml` for its own tests.
 
 ### Toolchain
 

pyproject.toml

@@ -0,0 +1,18 @@
+# Config only (no [project]/[build-system]/uv.lock) - the Scripts profile, CODESTYLE.md "Two profiles".
+
+[tool.ruff]
+target-version = "py313"
+line-length = 100
+
+[tool.ruff.lint]
+extend-select = ["I"]  # isort import ordering, on top of the default rules
+
+[tool.mypy]
+python_version = "3.13"
+files = ["spec", "host-setup"]
+
+[tool.pyright]
+pythonVersion = "3.13"
+typeCheckingMode = "standard"
+include = ["spec", "host-setup"]
+exclude = ["**/__pycache__"]

reports/divergences.md

@@ -0,0 +1,42 @@
+# Fleet divergence report
+
+Generated by `python3 spec/fidelity_honesty.py --report` - do not hand-edit. Curate dispositions in [`spec/divergences.json`][ledger] and regenerate. Each row reflects a repo's ground-truth branch at generation time. Git dates this file.
+
+## Burn-down
+
+### re-vendor
+
+- **.markdownlint-cli2.jsonc** - AudioCleaner, PhotoCleaner, aiopurpleair - Verbatim config held as a hand-modified copy in these three (not a past hub revision). Restore the current canonical.
+
+### upstream-candidate
+
+- **repo-config/configure.sh** - NxWitness - Not stale - a forked design: repo-specialized (Docker Hub image list, secret names, Make/Matrix.json product matrix) and adds a check/5D-audit mode the hub canonical lacks. Overwriting would regress it. Reconcile by adopting the check mode into the hub canonical, then re-vendoring.
+
+### investigate
+
+- **repo-config/configure.sh** - ESPHome-NonRoot, LanguageTags, VSCode-Server-DotNetCore, aiopurpleair, homeassistant-purpleair - Diverge from the 119-line apply-only canonical in a non-stale way, likely carrying the older check-mode design (same family as the NxWitness fork). Triage per repo: fold into the upstream-candidate decision, or re-vendor if merely a forked-then-abandoned copy.
+- **pyproject.toml** (manifest gap) - The hub gained a config-only Scripts-profile pyproject.toml in #388. Decide whether to track it (intent, appliesTo python) after confirming the python repos carry an equivalent.
+- **.github/workflows/publish-release.yml** (manifest gap) - Carried by some repos, absent from others, and varies widely (12 divergent, 5 absent). Needs a fidelity call (interface vs intent) and an appliesTo scope before tracking - it would surface many new findings.
+- **.github/workflows/validate-task.yml** (manifest gap) - As publish-release.yml (11 divergent, 9 absent): fidelity plus appliesTo decision pending.
+
+### accepted
+
+- **.editorconfig-checker.json** - HolidayLights, HomeAutomation-Config - Both carry a legitimate repo-specific Exclude list (HomeAutomation-Config excludes a Vantage/ subtree, HolidayLights excludes .fseq sequence files). The uniform Disable block is carried intent-equivalent. Exclude is inherently repo-local, which is why the unit is intent, not verbatim.
+- **LICENSE** (manifest gap) - Each repo owns its license file. The hub does not standardize license text, so it is intentionally outside the manifest.
+
+## Untriaged - add a disposition to `spec/divergences.json`
+
+_None - every live divergence has a recorded disposition._
+
+## Mechanical re-vendor (verbatim stale copies)
+
+A past hub revision, not the current canonical - the audit already flags these as DRIFT. Copy the current file down. No judgment needed.
+
+- **.markdownlint-cli2.jsonc** (15): DevKitCIoT, ESPHome-Config, ESPHome-NonRoot, HolidayLights, HomeAssistant-Config, HomeAutomation-Config, KiCadLibrary, LanguageTags, MediaTools, NxWitness, PlexCleaner, Utilities, VSCode-Server-DotNetCore, Vantage-Config, homeassistant-purpleair
+- **repo-config/configure.sh** (6): ESPHome-Config, HomeAssistant-Config, HomeAutomation-Config, PlexCleaner, Utilities, Vantage-Config
+
+## Promote candidates (intent uniform -> verbatim)
+
+_None - no intent unit is currently fleet-uniform with the canonical._
+
+[ledger]: ../spec/divergences.json

spec/audit.py

@@ -26,6 +26,7 @@
 import subprocess
 import sys
 from datetime import datetime, timezone
+from typing import Any
 
 ROOT = pathlib.Path(__file__).resolve().parent.parent
 
@@ -59,8 +60,8 @@ def hub_name():
 HUB_NAME, HUB_NAME_FROM_REMOTE = hub_name()
 
 
-def gh(path, ok404=False):
-    """GET a REST path via gh, returning parsed JSON or None on 404 when ok404.
+def gh(path, ok404=False) -> Any:
+    """GET a REST path via gh, returning parsed JSON, or None on a 404 (when ok404) or an empty response body.
 
     No --paginate: on object endpoints it concatenates page documents into unparseable JSON. Every
     list read here fits one page; callers pass per_page=100 where a default page could truncate.
@@ -276,7 +277,7 @@ def classify_verbatim(down_text, canon_text, past_texts):
     return "modified"
 
 
-_HISTORY_CACHE = {}  # rel_path -> [past revision content], reused as a canonical is compared against every audited repo
+_HISTORY_CACHE: dict[str, list[str]] = {}  # rel_path -> past revision contents, cached because one canonical is compared against every audited repo
 
 
 def git_file_history(rel_path):
@@ -506,7 +507,7 @@ def audit_repo(entry, spec):
         # to ""), whereas a too-large or non-inline payload returns encoding "none" (text stays None -> flagged).
         text = base64.b64decode(content["content"]).decode("utf-8", "replace") if content.get("encoding") == "base64" else None
         # Interface conformance (name + wiring) plus any verbatim job regions the contract pins.
-        if fid == "interface":
+        if item is not None and fid == "interface":
             if text is None:
                 findings.append(("DRIFT", f"interface: could not read {path} content on {ground} to verify its contract (no inline content returned); verify by hand"))
             else:
@@ -517,7 +518,7 @@ def audit_repo(entry, spec):
                     findings.extend(check_verbatim(f"{path} job '{job}'", text, canonical_rel,
                                                    extract=lambda t, j=job: split_jobs(t).get(j)))
         # Whole-file verbatim: byte-identical to the hub's canonical after EOL normalization.
-        elif fid == "verbatim":
+        elif item is not None and fid == "verbatim":
             if text is None:
                 findings.append(("DRIFT", f"verbatim: could not read {path} content on {ground} to compare (no inline content returned); verify by hand"))
             else:

spec/divergences.json

@@ -0,0 +1,16 @@
+{
+    "$schema": "./divergences.schema.json",
+    "note": "Curated dispositions for known fleet divergences from the manifest canonicals - the burn-down ledger. spec/fidelity_honesty.py --report joins this against live fleet reality to write reports/divergences.md. A recorded divergence still present renders as a burn-down task with its disposition. A live divergence absent here renders as UNTRIAGED. A recorded divergence no longer live renders as resolved. Edit this file (not the generated report) and regenerate. dispositions cover per-repo file divergences from a verbatim or intent canonical. gaps cover files carried by the fleet but absent from spec/files.json. disposition vocabulary: re-vendor (drift-to-fix, copy the current canonical down), track (a gap to add to the manifest), accepted (a legitimate permanent divergence, no action), upstream-candidate (the downstream carries an improvement the hub should adopt, then re-vendor), investigate (recorded, decision pending).",
+    "dispositions": [
+        { "path": ".editorconfig-checker.json", "repos": ["HomeAutomation-Config", "HolidayLights"], "disposition": "accepted", "reason": "Both carry a legitimate repo-specific Exclude list (HomeAutomation-Config excludes a Vantage/ subtree, HolidayLights excludes .fseq sequence files). The uniform Disable block is carried intent-equivalent. Exclude is inherently repo-local, which is why the unit is intent, not verbatim.", "tracking": null },
+        { "path": ".markdownlint-cli2.jsonc", "repos": ["aiopurpleair", "PhotoCleaner", "AudioCleaner"], "disposition": "re-vendor", "reason": "Verbatim config held as a hand-modified copy in these three (not a past hub revision). Restore the current canonical.", "tracking": null },
+        { "path": "repo-config/configure.sh", "repos": ["NxWitness"], "disposition": "upstream-candidate", "reason": "Not stale - a forked design: repo-specialized (Docker Hub image list, secret names, Make/Matrix.json product matrix) and adds a check/5D-audit mode the hub canonical lacks. Overwriting would regress it. Reconcile by adopting the check mode into the hub canonical, then re-vendoring.", "tracking": null },
+        { "path": "repo-config/configure.sh", "repos": ["LanguageTags", "aiopurpleair", "homeassistant-purpleair", "ESPHome-NonRoot", "VSCode-Server-DotNetCore"], "disposition": "investigate", "reason": "Diverge from the 119-line apply-only canonical in a non-stale way, likely carrying the older check-mode design (same family as the NxWitness fork). Triage per repo: fold into the upstream-candidate decision, or re-vendor if merely a forked-then-abandoned copy.", "tracking": null }
+    ],
+    "gaps": [
+        { "path": "LICENSE", "disposition": "accepted", "reason": "Each repo owns its license file. The hub does not standardize license text, so it is intentionally outside the manifest.", "tracking": null },
+        { "path": "pyproject.toml", "disposition": "investigate", "reason": "The hub gained a config-only Scripts-profile pyproject.toml in #388. Decide whether to track it (intent, appliesTo python) after confirming the python repos carry an equivalent.", "tracking": null },
+        { "path": ".github/workflows/publish-release.yml", "disposition": "investigate", "reason": "Carried by some repos, absent from others, and varies widely (12 divergent, 5 absent). Needs a fidelity call (interface vs intent) and an appliesTo scope before tracking - it would surface many new findings.", "tracking": null },
+        { "path": ".github/workflows/validate-task.yml", "disposition": "investigate", "reason": "As publish-release.yml (11 divergent, 9 absent): fidelity plus appliesTo decision pending.", "tracking": null }
+    ]
+}

spec/divergences.schema.json

@@ -0,0 +1,40 @@
+{
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
+    "$id": "https://github.com/ptr727/ProjectTemplate/spec/divergences.schema.json",
+    "title": "Fleet divergence dispositions (burn-down ledger)",
+    "type": "object",
+    "additionalProperties": false,
+    "properties": {
+        "$schema": { "type": "string" },
+        "note": { "type": "string" },
+        "dispositions": {
+            "type": "array",
+            "items": {
+                "type": "object",
+                "required": ["path", "repos", "disposition", "reason"],
+                "additionalProperties": false,
+                "properties": {
+                    "path": { "type": "string" },
+                    "repos": { "type": "array", "items": { "type": "string" }, "minItems": 1 },
+                    "disposition": { "enum": ["re-vendor", "track", "accepted", "upstream-candidate", "investigate"] },
+                    "reason": { "type": "string" },
+                    "tracking": { "type": ["string", "null"] }
+                }
+            }
+        },
+        "gaps": {
+            "type": "array",
+            "items": {
+                "type": "object",
+                "required": ["path", "disposition", "reason"],
+                "additionalProperties": false,
+                "properties": {
+                    "path": { "type": "string" },
+                    "disposition": { "enum": ["re-vendor", "track", "accepted", "upstream-candidate", "investigate"] },
+                    "reason": { "type": "string" },
+                    "tracking": { "type": ["string", "null"] }
+                }
+            }
+        }
+    }
+}