Fail on media the site carries but links from nowhere

TODO.md

@@ -51,6 +51,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no
 ## Open decisions
 
 - `/robots.txt/` and `/osd.xml/` currently sit in `slugs.map` pointing at `/`. The first would be better pointing at the real `/robots.txt`.
+- **The conversion dropped images from at least five galleries, and 120 carried media files are linked from no page.** Five `gallery` shortcodes are empty, across three posts: [`esp32-water-and-gas-utility-meter.md:25`](./content/posts/2021/08/09/esp32-water-and-gas-utility-meter.md), [`installing-flair-smart-vents-to-keep-room-temperatures-balanced.md:59,96,154`](./content/posts/2022/10/30/installing-flair-smart-vents-to-keep-room-temperatures-balanced.md), and [`halloween-pumpkins-and-lights.md:28`](./content/posts/2022/10/31/halloween-pumpkins-and-lights.md). `gallery` is the only shortcode this happens to. The ESP32 post is the clearest case: it says "Below are pictures of the finished case and the utility meters:" and then renders an empty div, while a sequential run of camera originals from that month sits on disk linked from nothing. Do not delete the empty shortcodes, which would erase the evidence and leave the prose promising pictures that never arrive. Recovering them needs the source export, since gallery membership and order are not derivable from what is carried here. `ORPHANED_MEDIA` in [`checks/check-url-parity.py`](./checks/check-url-parity.py) holds the count and falls as galleries are restored. The 120 are not all gallery losses; the remainder is unadjudicated and may include uploads the old platform never published.
 - Content is capped at a fixed 720px on every screen, because PaperMod's width is four CSS variables with no responsive term and no Hugo parameter. The prose measure is right and should stay; images and galleries inheriting the same cap is the part that costs something on a wide display. The knobs, the override location, and the `--gap` trap are documented under "Customization points" in [`themes/README.md`](./themes/README.md).
 
 ## Deliberate deviations from the fleet baseline

checks/README.md

@@ -10,7 +10,7 @@ The contract is enforced by two gates, because one cannot cover both halves:
 
 | Gate | Proves | Runs |
 | --- | --- | --- |
-| [`check-url-parity.py`](./check-url-parity.py) | Every URL that must render exists as a built page, every legacy image URL resolves, every local asset reference points at a real file | Against `public/`, in CI and before any release is installed |
+| [`check-url-parity.py`](./check-url-parity.py) | Every URL that must render exists as a built page, every legacy image URL resolves, every local asset reference points at a real file, and the count of carried media linked from no page still equals its recorded baseline | Against `public/`, in CI and before any release is installed |
 | [`check-live-urls.sh`](./check-live-urls.sh) | Every redirect resolves, and its destination answers | Against a running server, which is the only thing that exercises a redirect |
 
 ## The two lists
@@ -73,3 +73,9 @@ wc -l checks/golden-urls.txt checks/redirect-urls.txt checks/golden-media-legacy
 ## Directionality
 
 The parity check fails on a **missing** URL and only notes an **extra** one. New posts, new tags, and deeper pagination legitimately add URLs, and nothing legitimately removes a URL the site has served. That asymmetry is what makes the lists append-only, which in turn is what makes the length-floor assertion in `check-live-urls.sh` sound: without it a truncated list would make every assertion below it pass vacuously.
+
+Media is the one surface checked in **both** directions, and it has to be, because each direction is blind to the other's failure. Outward from a reference, the legacy list proves an inbound link still lands and the asset check proves a reference names a real file. Neither asks whether anything points at a given file, so an image dropped from a page during the conversion stays on disk, stays reachable at its own URL, and reports green in both directions while appearing nowhere on the site. The orphan check reads inward from the file and is the only one that sees it.
+
+Its constant is an exact count rather than a bound. A ceiling would let a drop leave slack behind for a later regression to hide in, so whatever lowers the count lowers the constant in the same change, and the check names the new number when it drops.
+
+A count is all the check can observe, and two causes reach each direction: it rises when a page stops linking media **or** when unlinked media is added, and it falls when media is linked from a page **or** when orphaned files are deleted. The messages name both, because naming one would send a reader looking for a page that never changed.

checks/check-url-parity.py

@@ -1,9 +1,10 @@
 #!/usr/bin/env python3
 """Verify the built site against the URL contract.
 
-Checks that every URL which must render exists, that every legacy media URL resolves, and that
-every local asset reference points at a file. Redirects need a running server and are checked
-by check-live-urls.sh instead.
+Checks that every URL which must render exists, that every legacy media URL resolves, that
+every local asset reference points at a file, and that the number of carried media files
+linked from no page still equals its recorded baseline. Redirects need a running server and
+are checked by check-live-urls.sh instead.
 """
 
 import pathlib
@@ -21,6 +22,17 @@
 
 CHECKS = pathlib.Path(__file__).resolve().parent
 
+# Media carried by the import that no built page links to.
+# The other two media checks run outward from a reference and cannot see these: a legacy URL
+# resolving proves an inbound link still lands, and a reference resolving proves it names a real
+# file. Neither asks whether anything points at a given file, so an image the conversion dropped
+# from a page stays reachable by URL, invisible on the site, and green in both directions.
+# Every one traces to the WordPress conversion rather than to anything this repo does. Five empty
+# gallery shortcodes across three posts are the identified cause of some of them, and the rest are
+# unadjudicated. The count is exact rather than a bound, so restoring a gallery lowers it here in
+# the same change and slack can never accumulate for a later regression to hide in.
+ORPHANED_MEDIA = 120
+
 
 def load(name):
     lines = [ln.strip() for ln in (CHECKS / name).read_text().splitlines()]
@@ -69,42 +81,101 @@ def check_media(public):
     return missing
 
 
-def check_assets(public):
-    """Check that every local asset a built page references exists on disk.
+def collect_refs(public):
+    """Every local asset reference in the built pages.
 
-    Catches a media file renamed, dropped, or never localized.
+    Read once and shared, since the assets and orphans checks are the same reference set
+    read in opposite directions.
     """
     # Minification drops the quotes around an attribute value that does not need them.
     # Matching only the quoted form checks a fraction of the references and calls it a pass.
     quoted = re.compile(r'(?:src|href|srcset)="(/(?:media|external)/[^"]+)"')
     bare = re.compile(r"(?:src|href|srcset)=(/(?:media|external)/[^\s\"'>]+)")
-    refs, missing = set(), []
+    refs = set()
     for page in public.rglob("*.html"):
         text = page.read_text(encoding="utf-8", errors="ignore")
         refs.update(quoted.findall(text))
         refs.update(bare.findall(text))
-    for ref in sorted(refs):
-        # Imported references carry resize parameters a static file server ignores.
-        # Some also escape an underscore, which a server decodes before looking up the file.
-        path = unquote(ref.split("?", 1)[0].split("#", 1)[0])
-        if not (public / path.lstrip("/")).is_file():
-            missing.append(ref)
+    return refs
+
+
+def ref_to_path(ref):
+    """Map a reference to the path under the built site it names."""
+    # Imported references carry resize parameters a static file server ignores.
+    # Some also escape an underscore, which a server decodes before looking up the file.
+    return unquote(ref.split("?", 1)[0].split("#", 1)[0]).lstrip("/")
+
+
+def check_assets(public, refs):
+    """Check that every local asset a built page references exists on disk.
+
+    Catches a media file renamed, dropped, or never localized.
+    """
+    missing = [ref for ref in sorted(refs) if not (public / ref_to_path(ref)).is_file()]
     print(f"assets : {len(refs) - len(missing)}/{len(refs)} local asset references resolve")
     return missing
 
 
+def check_orphans(public, refs):
+    """Check that every carried media file is linked from some built page.
+
+    The reverse of the assets check, and the only one that can see an image the conversion
+    dropped from a page: it stays on disk and reachable by URL, so nothing else objects.
+    """
+    linked = {ref_to_path(ref) for ref in refs}
+    carried, orphaned = 0, []
+    for tree in ("media", "external"):
+        root = public / tree
+        if not root.is_dir():
+            continue
+        for path in root.rglob("*"):
+            if not path.is_file():
+                continue
+            carried += 1
+            # `linked` holds URL paths, which are always forward-slashed, so a native separator
+            # here would match nothing and report every carried file as an orphan. check_render
+            # normalizes for the same reason.
+            rel = str(path.relative_to(public)).replace("\\", "/")
+            if rel not in linked:
+                orphaned.append(rel)
+    orphaned.sort()
+    # No media at all is a broken build, not progress. Left to the comparison below it reads as
+    # zero orphans, which is fewer than the baseline, and the advice would be to lower
+    # ORPHANED_MEDIA to 0 - a gate talking the reader into switching it off.
+    if carried == 0:
+        print("orphans: no media files in the built site - the output is incomplete or mislocated")
+        return ["public/media and public/external are both absent or empty"]
+    print(f"orphans: {len(orphaned)} of {carried} carried media files are linked from no page")
+    if len(orphaned) == ORPHANED_MEDIA:
+        return []
+    # The explanation is printed rather than returned, so the caller's count stays the orphan
+    # count. A diagnostic carried in the failure list would make the reported total one too many.
+    # A count is all this can observe, and two causes reach each direction. Naming one of them
+    # would send a reader looking for a page that never changed.
+    if len(orphaned) > ORPHANED_MEDIA:
+        print(f"         expected {ORPHANED_MEDIA} - a page stopped linking media, or unlinked media was added")
+        return orphaned
+    print(
+        f"         expected {ORPHANED_MEDIA} - media was linked from a page, or orphaned files were "
+        f"removed; lower ORPHANED_MEDIA to {len(orphaned)} in this change rather than leaving the slack"
+    )
+    return orphaned
+
+
 def main(argv):
     if len(argv) != 2:
         sys.exit(f"usage: {argv[0]} <public-dir>")
     public = pathlib.Path(argv[1])
     if not public.is_dir():
         sys.exit(f"FAIL: {public} is not a directory - run hugo first")
 
+    refs = collect_refs(public)
     failures = []
     for label, missing in (
         ("render", check_render(public)),
         ("media", check_media(public)),
-        ("assets", check_assets(public)),
+        ("assets", check_assets(public, refs)),
+        ("orphans", check_orphans(public, refs)),
     ):
         if missing:
             failures.append((label, missing))