From 7886e89a81f468469619bc39c22988c4dfa327fb Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 07:25:35 -0700 Subject: [PATCH 1/7] Emit robots.txt, and gate the base URL it is derived from The old platform serves a robots.txt today and this site emitted none, so the cutover was not a return to a previous state, it was a move from having crawl directives to having none on a site that has had them for years. Hugo emits the file only when enableRobotsTXT is set, which is why /robots.txt answered 404 through the first production deploy. The Sitemap: line is the load-bearing part rather than any rule. Across the interim hostname's first full day no crawler fetched sitemap.xml or feed.xml once, every request to either coming from curl: a crawler is told where a sitemap is rather than guessing, and the only thing telling them is the file the cutover deletes. Every Disallow the old platform serves names a WordPress path this site does not have, so the content that preserves today's behavior is User-agent: *, no Disallow, and the sitemap. The theme's template already derives that URL from the built baseURL, so setting the flag is the whole change and there is no second edit to remember at the cutover. Verified across two base URLs: the mirror advertises its own sitemap and a production build advertises the site's. /robots.txt/ with a trailing slash is in the redirect contract and sent visitors to the home page. It now resolves to the real file, fixed in build-redirects.py rather than in the generated map, since the map is rewritten from the capture and a hand edit does not survive. Verified byte-identical regeneration otherwise: the map diff is that one line. /osd.xml/ stays pointed at the home page, being an OpenSearch description this site does not emit. The check is worth more than the file. Every contract list is path-only and check-live-urls.sh joins whatever base it is handed, which is what lets one contract cover four environments and also means a build baked with the wrong host passes all 1,245 URLs while every canonical tag and sitemap entry names another site. robots.txt is the one artifact whose absolute URL a gate can read without being told what to expect, so the parity check compares it against the origin read from the home page's canonical link. Four failures gated, all four demonstrated failing before the check was trusted: file absent, no Sitemap line, a line naming another origin, and a line advertising a sitemap not built. Released to the local production mirror and checked live, since this touches deploy/maps and hugo.yaml: 1245 URLs honored, /robots.txt/ 301s to /robots.txt, and that file answers 200 as text/plain. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 11 ++++++----- checks/README.md | 10 ++++++++++ checks/build-redirects.py | 13 ++++++++++++- checks/check-url-parity.py | 40 +++++++++++++++++++++++++++++++++++++- deploy/maps/slugs.map | 2 +- hugo.yaml | 8 ++++++++ 6 files changed, 76 insertions(+), 8 deletions(-) diff --git a/TODO.md b/TODO.md index a8cbb5b..6dc0ad8 100644 --- a/TODO.md +++ b/TODO.md @@ -33,11 +33,12 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. -- **Decide what `robots.txt` says, now before the cutover rather than before the first deploy.** Production answers 404 for it, so `X-Robots-Tag` is the only thing keeping the interim hostname out of an index. Crawlers are already asking: OAI-SearchBot, ClaudeBot, GPTBot, and Scrapy each arrived within three hours of the certificate being issued, at a hostname with no inbound links. - - **The deploy did not fix the 404, which is the part that reads the wrong way round, and is now measured rather than predicted.** The VPS agent raised this in §22.10 and asked two questions of this side. Both are now measured rather than assumed: **this site emits no `robots.txt` at all**, because `hugo.yaml` sets no `enableRobotsTXT`, so the 404 survives the deploy and `X-Robots-Tag` stays the only control. **`sitemap.xml` is emitted** and becomes fetchable on the interim name at that same deploy, listing whatever `HUGO_BASEURL` was built with, which is the second reason that variable mattered. Read together: after the rehearsal deploy a crawler gets a full sitemap and no robots file. - - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Since this site emits none, M7b is not a return to a previous state, it is a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer goes with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the `.com` sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **Deliberately undecided**, and the three options are to preserve today's behavior, to write what this site actually wants, or to keep emitting nothing and accept the loss. +- **`robots.txt` is decided and built, 2026-08-08, and what remains is that production has not been redeployed since.** The site emits one now, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. The record below is kept because the reasoning is what the next decision about crawl directives will need, and because production still answers 404 until a deploy carries this. + - **The first deploy did not fix the 404, and that is what turned this from a gap into a decision.** The VPS agent raised it in §22.10 and both halves were measured rather than assumed: the site emitted no `robots.txt` at all, because `hugo.yaml` set no `enableRobotsTXT`, so the 404 survived the deploy and `X-Robots-Tag` was the only control, while `sitemap.xml` **was** emitted and became fetchable on the interim name at that same deploy. A crawler got a full sitemap and no robots file. `enableRobotsTXT` is now set, so this describes the release production is still serving rather than the current build. + - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Since this site emits none, M7b is not a return to a previous state, it is a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer goes with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **That is what was chosen**, out of three options: preserve today's behavior, write what this site actually wants, or keep emitting nothing and accept the loss. The sitemap URL is derived from the built `baseURL` rather than typed, which is what makes the choice survive the cutover without a second edit. - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel. - - **`/robots.txt/`, with a trailing slash, is an entry in `slugs.map` pointing at `/`**, noted under "Open decisions" below. If a real file lands, that entry points at it instead, and the change ships in the same release rather than after it. + - **`/robots.txt/`, with a trailing slash, now redirects to the real file** rather than to the home page, in the same change, since the two are only correct together. The fix is in `build-redirects.py` rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. `/osd.xml/` stays pointed at the home page: it was the old platform's OpenSearch description and this site emits no such file. + - **A wrong `HUGO_BASEURL` is now visible to a gate here, which it was not before.** Every contract list is path-only and the live check joins whatever base it is given, so a build baked with the wrong host passes all 1,245 URLs; the host side found the same blind spot in its own smoke test and had to read the deployed bytes by hand. `robots.txt` carries an absolute URL derived from the same `baseURL`, and the parity check compares it against the origin read from the home page's canonical link, so the two must agree. That is a side effect of this change rather than its purpose, and it is the more durable half. - **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. - **Restore file mtimes in CI so `--link-dest` links, and do it after the media check rather than before.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. @@ -72,7 +73,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no - **The same question covers the migration toolchain in the capture directory**, which is fourteen scripts: the `wp2hugo` run, the content restructure and clean passes, external-media localization, the crawl and mirror, the golden-URL build, and the media inventory. Some are worth keeping only if generalized, and some are cheaper to rewrite than to maintain, so this is a per-script call rather than one decision. - **One of them is already three copies with two of them stale**, which is the concrete version of this risk rather than a hypothetical one. `build-redirects.py` exists at the capture root, again under the capture's own `checks/`, and here at [`checks/build-redirects.py`](./checks/build-redirects.py). The two capture copies are identical to each other at 115 lines; the copy in this repository is the maintained one at 225. Nothing detects that, because the capture is not a git repository and is read-only in normal use. - **What `robots.txt` says, which is undecided and is the last non-mechanical item before M7b.** Recorded under "Next" above, where it blocks the cutover. -- `/robots.txt/` and `/osd.xml/` currently sit in `slugs.map` pointing at `/`. The first would be better pointing at the real `/robots.txt`. +- `/osd.xml/` sits in `slugs.map` pointing at `/`, and stays there unless this site ever emits an OpenSearch description. `/robots.txt/` was the other half and is resolved, above. - Content is capped at a fixed 720px on every screen, because PaperMod's width is four CSS variables with no responsive term and no Hugo parameter. The prose measure is right and should stay; images and galleries inheriting the same cap is the part that costs something on a wide display. The knobs, the override location, and the `--gap` trap are documented under "Customization points" in [`themes/README.md`](./themes/README.md). ## Deliberate deviations from the fleet baseline diff --git a/checks/README.md b/checks/README.md index 83bc682..3c1e009 100644 --- a/checks/README.md +++ b/checks/README.md @@ -84,6 +84,16 @@ A count is all the check can observe, and two causes reach each direction: it ri **Both directions read absolute references as well as relative ones.** Hugo writes an absolute URL wherever a template resolves one against the base, which the entry-cover image on every list page does. Reading only rooted paths made those files look linked from nowhere while they were being displayed, and left a broken one unchecked in the other direction. The origin is read from the home page's canonical link rather than assumed, since staging and production build with different base URLs and a hardcoded host would check one environment's output against another's. No canonical link is a hard failure, because a guessed origin inflates the orphan count by exactly the pages that use one. +## The robots check, which is the only place a wrong base URL is visible + +Every list here is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate: it is what lets one contract be run against four environments without editing a line. The cost is that **a build baked with the wrong host passes the entire contract.** Point the check at the site it just built and all 1,245 URLs answer correctly while every canonical tag, feed entry, and sitemap entry names a different site. The host side found the same blind spot in its own smoke test and reported it, having checked the deployed bytes by hand instead. + +`robots.txt` is the one artifact that carries an absolute URL the gate can read without being told what to expect, because its `Sitemap:` line is derived from the same `baseURL` everything else is, and the origin it must match is read from the home page's canonical link rather than configured. So the check compares the build against itself, and a `HUGO_BASEURL` naming the wrong environment fails here rather than at a reader. + +The file is worth asserting for its own sake as well. Hugo emits none unless `enableRobotsTXT` is set, which is why this site answered 404 for a file the old platform served, and the `Sitemap:` line is load-bearing in a way the crawl rules are not: across the interim hostname's first day no crawler fetched the sitemap once, because a crawler is told where a sitemap is rather than guessing it. Four failures are gated — the file absent, no `Sitemap:` line, a line naming another origin, and a line advertising a sitemap that was not built — and all four were demonstrated failing before the check was trusted. + +`/robots.txt/`, with a trailing slash, is a URL the old platform served and is in the redirect contract. It resolves through `slugs.map` like any other one-segment legacy URL, and the generator special-cases it to the real file rather than to the home page. `/osd.xml/` stays pointed at the home page deliberately: it was the old platform's OpenSearch description and this site emits no such file. + ## The gallery check, which no direction above can reach Every check above reasons about a URL: whether it renders, whether it resolves, whether anything points at it. Content misplaced **inside** a gallery satisfies all of that. The file exists, the reference resolves, and something links it, so the media surface is green in both directions while the page is laid out wrong. The defect is one of structure, which is why three variants of it survived the conversion and every gate since. diff --git a/checks/build-redirects.py b/checks/build-redirects.py index 414ef9f..3d94171 100755 --- a/checks/build-redirects.py +++ b/checks/build-redirects.py @@ -19,6 +19,15 @@ # The WordPress importer registers the full slug, and both answer, so both are mapped. BLOGGER_SLUG_LIMIT = 40 +# A one-segment URL naming a file the site serves at the root, rather than a page slug. +# The resolver below reads these as unresolvable attachment slugs and sends them to the home +# page, which is the right answer for a slug nothing claims and the wrong one for a file that +# exists: /robots.txt/ should reach /robots.txt. Named here rather than hand-edited into the +# generated map, because the map is rewritten from the capture and a hand edit does not survive. +# /osd.xml/ stays out deliberately. It was the old platform's OpenSearch description and this +# site emits no such file, so the home page remains the honest destination for it. +WELL_KNOWN = {"/robots.txt/": "/robots.txt"} + def text(el, path): node = el.find(path, NS) @@ -179,7 +188,9 @@ def main(argv): resolved, via_parent, via_media, orphan = [], 0, 0, [] for u in needed: slug = u.strip("/") - if slug in by_slug: + if u in WELL_KNOWN: + resolved.append((u, WELL_KNOWN[u])) + elif slug in by_slug: resolved.append((u, by_slug[slug])) via_parent += 1 elif slug.lower() in file_to_post: diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index 2d017fa..1e7e949 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -42,7 +42,7 @@ # what a URL that did not build is, and it is not what a stray node inside a gallery is: those are # present, which is the whole complaint. The default stays "missing" so a check added later reads # the way the older ones do unless it says otherwise. -FAILURE_NOUN = {"gallery": "stray nodes"} +FAILURE_NOUN = {"gallery": "stray nodes", "robots": "problems"} def load(name): @@ -92,6 +92,43 @@ def check_media(public): return missing +def check_robots(public): + """Check that robots.txt exists and advertises this build's own sitemap. + + Two failures, one gate. Hugo emits no robots.txt unless enableRobotsTXT is set, which is how + this site answered 404 for years, and the file's only load-bearing line points at the sitemap + a crawler would otherwise never find. + + The sitemap URL is absolute and derived from baseURL, which makes this the one build-time + assertion that can see a wrong baseURL at all. Every contract list is path-only and the live + check joins whatever base it is given, so a build baked with the wrong host passes all of them + while every canonical tag, feed entry and sitemap entry names the wrong site. + """ + robots = public / "robots.txt" + if not robots.is_file(): + print("robots : missing") + return ["robots.txt was not built - enableRobotsTXT is off in hugo.yaml"] + + origin = site_origin(public) + advertised = re.findall(r"(?mi)^\s*Sitemap:\s*(\S+)\s*$", robots.read_text(encoding="utf-8")) + if not advertised: + print("robots : built, no Sitemap line") + return ["robots.txt carries no Sitemap: line - a crawler will not find the sitemap unaided"] + + wrong = [u for u in advertised if not u.startswith(origin + "/")] + if wrong: + print(f"robots : built, {len(wrong)} Sitemap line(s) naming another origin") + return [f"{u} (this build's origin is {origin})" for u in wrong] + + # Named rather than counted, since there is one today and the line is the thing being checked. + print(f"robots : built, advertising {advertised[0]}") + return [ + f"{u} (advertised, but {u[len(origin) + 1:]} was not built)" + for u in advertised + if not (public / u[len(origin) + 1 :]).is_file() + ] + + def site_origin(public): """The site's own scheme and host, read from the artifact rather than assumed. @@ -307,6 +344,7 @@ def main(argv): ("assets", check_assets(public, refs)), ("orphans", check_orphans(public, refs)), ("gallery", check_galleries(public)), + ("robots", check_robots(public)), ): if found: failures.append((label, found)) diff --git a/deploy/maps/slugs.map b/deploy/maps/slugs.map index d27247c..2eb63c0 100644 --- a/deploy/maps/slugs.map +++ b/deploy/maps/slugs.map @@ -100,7 +100,7 @@ /jellyfish-lighting-ports-500x214-1/ / /osd.xml/ / /packet/ /2020/09/05/abandoning-greeneye-for-sense/ -/robots.txt/ / +/robots.txt/ /robots.txt /sce_smartmeter/ /2020/09/05/abandoning-greeneye-for-sense/ /ted-dashboard/ /2020/09/05/abandoning-greeneye-for-sense/ /ted-graphing/ /2020/09/05/abandoning-greeneye-for-sense/ diff --git a/hugo.yaml b/hugo.yaml index 2ec8d0f..6f5e3e8 100644 --- a/hugo.yaml +++ b/hugo.yaml @@ -27,6 +27,14 @@ refLinksErrorLevel: ERROR # The deploy rebuilds the site rather than receiving it as an artifact, so the build stays reproducible. enableGitInfo: false +# Off by default in Hugo, which is why this site answered 404 for a file the old platform served. +# The Sitemap: line is the load-bearing part: no crawler has ever fetched this site's sitemap +# unprompted, because a crawler is told where a sitemap is rather than guessing, and the only +# thing telling them today is the robots.txt the cutover deletes. +# The theme's template derives that URL from the built baseURL, so it names .net during the +# rehearsal and .com afterwards with no second edit to remember at the cutover. +enableRobotsTXT: true + params: description: My discoveries, frustrations, and delights defaultTheme: auto From 33eaa0e0fb5e4ee5dcdc0390dc9fc9341e34e4f4 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 07:31:31 -0700 Subject: [PATCH 2/7] Bound both claims to what was measured Two overclaims in comments, both real. The hugo.yaml note said no crawler has ever fetched this site's sitemap unprompted, where the evidence is one day of traffic on one hostname, and the docstring said the site answered 404 for years, where the behaviour is a config flag rather than a duration. Each now says what was observed and over what window. Co-Authored-By: Claude Opus 5 (1M context) --- checks/check-url-parity.py | 6 +++--- hugo.yaml | 7 ++++--- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index 1e7e949..570029a 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -95,9 +95,9 @@ def check_media(public): def check_robots(public): """Check that robots.txt exists and advertises this build's own sitemap. - Two failures, one gate. Hugo emits no robots.txt unless enableRobotsTXT is set, which is how - this site answered 404 for years, and the file's only load-bearing line points at the sitemap - a crawler would otherwise never find. + Two failures, one gate. Hugo emits no robots.txt unless enableRobotsTXT is set, which is why + this site served none until that flag was turned on, and the file's only load-bearing line + points at the sitemap a crawler is otherwise unlikely to find, being told rather than guessing. The sitemap URL is absolute and derived from baseURL, which makes this the one build-time assertion that can see a wrong baseURL at all. Every contract list is path-only and the live diff --git a/hugo.yaml b/hugo.yaml index 6f5e3e8..ded53f7 100644 --- a/hugo.yaml +++ b/hugo.yaml @@ -28,9 +28,10 @@ refLinksErrorLevel: ERROR enableGitInfo: false # Off by default in Hugo, which is why this site answered 404 for a file the old platform served. -# The Sitemap: line is the load-bearing part: no crawler has ever fetched this site's sitemap -# unprompted, because a crawler is told where a sitemap is rather than guessing, and the only -# thing telling them today is the robots.txt the cutover deletes. +# The Sitemap: line is the load-bearing part: across the interim hostname's first full day of +# traffic, every request for sitemap.xml came from curl and none from a crawler, because a crawler +# is told where a sitemap is rather than guessing, and the only thing telling them today is the +# robots.txt the cutover deletes. # The theme's template derives that URL from the built baseURL, so it names .net during the # rehearsal and .com afterwards with no second edit to remember at the cutover. enableRobotsTXT: true From fcd81820d0f7ef7a75201b764c611c1fd26197cb Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 07:35:53 -0700 Subject: [PATCH 3/7] Withdraw the wrong-baseURL claim, which the check cannot support Four places said this check makes a wrong HUGO_BASEURL visible. It does not, and the reason is structural rather than a bug: the advertised sitemap origin and the canonical origin it is compared against are both derived from baseURL, so they agree whenever the build is coherent, including when baseURL held the wrong value for the environment being deployed to. Nothing inside the artifact can see that, which is exactly why the VPS side reads the origin out of the deployed sitemap.xml, og:url and feed.xml and reports the counts either way. Raised in review, in all four places at once, and correct. Each now states what the comparison proves, internal consistency, and what it still catches: an origin written rather than derived, where a committed static/robots.txt shadowing the template is how that happens, and a sitemap advertised but never built. A gate described as catching more than it catches is worse than no gate, because the next person stops looking for the failure it does not see. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 2 +- checks/README.md | 10 ++++++---- checks/check-url-parity.py | 12 ++++++++---- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/TODO.md b/TODO.md index 6dc0ad8..2ad6e2a 100644 --- a/TODO.md +++ b/TODO.md @@ -38,7 +38,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Since this site emits none, M7b is not a return to a previous state, it is a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer goes with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **That is what was chosen**, out of three options: preserve today's behavior, write what this site actually wants, or keep emitting nothing and accept the loss. The sitemap URL is derived from the built `baseURL` rather than typed, which is what makes the choice survive the cutover without a second edit. - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel. - **`/robots.txt/`, with a trailing slash, now redirects to the real file** rather than to the home page, in the same change, since the two are only correct together. The fix is in `build-redirects.py` rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. `/osd.xml/` stays pointed at the home page: it was the old platform's OpenSearch description and this site emits no such file. - - **A wrong `HUGO_BASEURL` is now visible to a gate here, which it was not before.** Every contract list is path-only and the live check joins whatever base it is given, so a build baked with the wrong host passes all 1,245 URLs; the host side found the same blind spot in its own smoke test and had to read the deployed bytes by hand. `robots.txt` carries an absolute URL derived from the same `baseURL`, and the parity check compares it against the origin read from the home page's canonical link, so the two must agree. That is a side effect of this change rather than its purpose, and it is the more durable half. + - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. - **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. - **Restore file mtimes in CI so `--link-dest` links, and do it after the media check rather than before.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. diff --git a/checks/README.md b/checks/README.md index 3c1e009..f8b627f 100644 --- a/checks/README.md +++ b/checks/README.md @@ -84,13 +84,15 @@ A count is all the check can observe, and two causes reach each direction: it ri **Both directions read absolute references as well as relative ones.** Hugo writes an absolute URL wherever a template resolves one against the base, which the entry-cover image on every list page does. Reading only rooted paths made those files look linked from nowhere while they were being displayed, and left a broken one unchecked in the other direction. The origin is read from the home page's canonical link rather than assumed, since staging and production build with different base URLs and a hardcoded host would check one environment's output against another's. No canonical link is a hard failure, because a guessed origin inflates the orphan count by exactly the pages that use one. -## The robots check, which is the only place a wrong base URL is visible +## The robots check, and the one thing no gate here can do -Every list here is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate: it is what lets one contract be run against four environments without editing a line. The cost is that **a build baked with the wrong host passes the entire contract.** Point the check at the site it just built and all 1,245 URLs answer correctly while every canonical tag, feed entry, and sitemap entry names a different site. The host side found the same blind spot in its own smoke test and reported it, having checked the deployed bytes by hand instead. +Hugo emits no `robots.txt` unless `enableRobotsTXT` is set, so this site served none until the flag was turned on, and the old platform serves one. The `Sitemap:` line is load-bearing in a way the crawl rules are not: across the interim hostname's first full day of traffic, every request for `sitemap.xml` came from `curl` and none from a crawler, because a crawler is told where a sitemap is rather than guessing it. Four failures are gated — the file absent, no `Sitemap:` line, a line naming another origin, and a line advertising a sitemap that was not built — and all four were demonstrated failing before the check was trusted. -`robots.txt` is the one artifact that carries an absolute URL the gate can read without being told what to expect, because its `Sitemap:` line is derived from the same `baseURL` everything else is, and the origin it must match is read from the home page's canonical link rather than configured. So the check compares the build against itself, and a `HUGO_BASEURL` naming the wrong environment fails here rather than at a reader. +**The `Sitemap:` line is the only absolute URL in the build a gate here can read.** Every list is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate and is what lets one contract cover four environments, so an origin is normally joined rather than checked. This is the one place one is compared. -The file is worth asserting for its own sake as well. Hugo emits none unless `enableRobotsTXT` is set, which is why this site answered 404 for a file the old platform served, and the `Sitemap:` line is load-bearing in a way the crawl rules are not: across the interim hostname's first day no crawler fetched the sitemap once, because a crawler is told where a sitemap is rather than guessing it. Four failures are gated — the file absent, no `Sitemap:` line, a line naming another origin, and a line advertising a sitemap that was not built — and all four were demonstrated failing before the check was trusted. +**What that comparison proves is internal consistency, and it is worth being exact about the limit.** The advertised origin must match the one read from the home page's canonical link, and both are derived from the same `baseURL`, so they agree whenever the build is coherent — including when `baseURL` held the wrong value for the environment being deployed to. **Nothing inside the artifact can detect that**, which is why the check belongs on the side that knows which host it is serving: the VPS side reads the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml` and reports the counts either way. A build baked with the wrong host still passes all 1,245 URLs here, and it passes this too. + +What the comparison does catch is an origin that was **written rather than derived** — a committed `static/robots.txt` shadowing the template is the way that happens, and pasting the old platform's `.com` sitemap line into one is the specific mistake it would catch — along with a sitemap advertised but never built. `/robots.txt/`, with a trailing slash, is a URL the old platform served and is in the redirect contract. It resolves through `slugs.map` like any other one-segment legacy URL, and the generator special-cases it to the real file rather than to the home page. `/osd.xml/` stays pointed at the home page deliberately: it was the old platform's OpenSearch description and this site emits no such file. diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index 570029a..877e2a7 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -99,10 +99,14 @@ def check_robots(public): this site served none until that flag was turned on, and the file's only load-bearing line points at the sitemap a crawler is otherwise unlikely to find, being told rather than guessing. - The sitemap URL is absolute and derived from baseURL, which makes this the one build-time - assertion that can see a wrong baseURL at all. Every contract list is path-only and the live - check joins whatever base it is given, so a build baked with the wrong host passes all of them - while every canonical tag, feed entry and sitemap entry names the wrong site. + The Sitemap: line is the only absolute URL in the build that a gate here can read at all, every + contract list being path-only, so it is the only place an origin is checked rather than joined. + What it proves is internal consistency: the advertised origin matches the one the home page's + canonical link carries. Both come from baseURL, so this cannot tell that baseURL was the wrong + value for the environment being deployed to - nothing in the artifact can, which is why that + check belongs on the side that knows which host it is serving. It does catch an origin that was + written rather than derived, a static robots.txt shadowing the template being the way that + happens, and it catches a sitemap advertised but not built. """ robots = public / "robots.txt" if not robots.is_file(): From e2da5a2de0f54d6aa4e3bc6058fd0bcb3bf79e47 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 07:42:25 -0700 Subject: [PATCH 4/7] Stop the robots check crashing on bytes it exists to inspect, and narrow two claims Three findings, all correct. read_text with strict UTF-8 raises out of the entire parity run on an invalid byte, so the one gate written to report a bad robots.txt would stack-trace on one instead of failing normally. Confirmed both ways: strict raises UnicodeDecodeError at position 67 on a file carrying 0xff, and errors=replace reports through the ordinary path. The file most likely to carry it is a committed static/robots.txt, which is the case this check is for. The docstring still said two failures where there are four, and both the docstring and the README said the Sitemap: line is the only absolute URL this gate can read. It is not, since site_origin reads the canonical link and collect_refs reads absolute asset references. The true and narrower point is that it is the only place an origin is compared rather than joined. Co-Authored-By: Claude Opus 5 (1M context) --- checks/README.md | 2 +- checks/check-url-parity.py | 25 +++++++++++++++++-------- 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/checks/README.md b/checks/README.md index f8b627f..89d866a 100644 --- a/checks/README.md +++ b/checks/README.md @@ -88,7 +88,7 @@ A count is all the check can observe, and two causes reach each direction: it ri Hugo emits no `robots.txt` unless `enableRobotsTXT` is set, so this site served none until the flag was turned on, and the old platform serves one. The `Sitemap:` line is load-bearing in a way the crawl rules are not: across the interim hostname's first full day of traffic, every request for `sitemap.xml` came from `curl` and none from a crawler, because a crawler is told where a sitemap is rather than guessing it. Four failures are gated — the file absent, no `Sitemap:` line, a line naming another origin, and a line advertising a sitemap that was not built — and all four were demonstrated failing before the check was trusted. -**The `Sitemap:` line is the only absolute URL in the build a gate here can read.** Every list is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate and is what lets one contract cover four environments, so an origin is normally joined rather than checked. This is the one place one is compared. +**The `Sitemap:` line is the one place this gate compares an origin rather than joining one.** Every list is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate and is what lets one contract cover four environments. Other absolute URLs are read here, the home page's canonical link and the absolute asset references described above among them, but they are read to resolve a reference rather than to check one host against another. **What that comparison proves is internal consistency, and it is worth being exact about the limit.** The advertised origin must match the one read from the home page's canonical link, and both are derived from the same `baseURL`, so they agree whenever the build is coherent — including when `baseURL` held the wrong value for the environment being deployed to. **Nothing inside the artifact can detect that**, which is why the check belongs on the side that knows which host it is serving: the VPS side reads the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml` and reports the counts either way. A build baked with the wrong host still passes all 1,245 URLs here, and it passes this too. diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index 877e2a7..b78b083 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -95,13 +95,18 @@ def check_media(public): def check_robots(public): """Check that robots.txt exists and advertises this build's own sitemap. - Two failures, one gate. Hugo emits no robots.txt unless enableRobotsTXT is set, which is why - this site served none until that flag was turned on, and the file's only load-bearing line - points at the sitemap a crawler is otherwise unlikely to find, being told rather than guessing. - - The Sitemap: line is the only absolute URL in the build that a gate here can read at all, every - contract list being path-only, so it is the only place an origin is checked rather than joined. - What it proves is internal consistency: the advertised origin matches the one the home page's + Four failures, one gate: the file absent, no Sitemap: line, a line naming another origin, and a + line advertising a sitemap that was not built. Hugo emits no robots.txt unless enableRobotsTXT + is set, which is why this site served none until that flag was turned on, and the file's only + load-bearing line points at the sitemap a crawler is otherwise unlikely to find, being told + rather than guessing. + + Every contract list is path-only and check-live-urls.sh joins whatever base URL it is handed, so + this is the only place the gate *compares* an origin rather than joining one. Other absolute + URLs are read here - the home page's canonical link, and absolute asset references - but they + are read to resolve a reference rather than to check a host against another. + + What the comparison proves is internal consistency: the advertised origin matches the one the canonical link carries. Both come from baseURL, so this cannot tell that baseURL was the wrong value for the environment being deployed to - nothing in the artifact can, which is why that check belongs on the side that knows which host it is serving. It does catch an origin that was @@ -114,7 +119,11 @@ def check_robots(public): return ["robots.txt was not built - enableRobotsTXT is off in hugo.yaml"] origin = site_origin(public) - advertised = re.findall(r"(?mi)^\s*Sitemap:\s*(\S+)\s*$", robots.read_text(encoding="utf-8")) + # errors="replace" rather than strict, or invalid UTF-8 raises out of the whole parity run and a + # gate that exists to report a bad robots.txt stack-traces on one instead. A committed + # static/robots.txt is the file most likely to carry it, and it is the case this check is for. + text = robots.read_text(encoding="utf-8", errors="replace") + advertised = re.findall(r"(?mi)^\s*Sitemap:\s*(\S+)\s*$", text) if not advertised: print("robots : built, no Sitemap line") return ["robots.txt carries no Sitemap: line - a crawler will not find the sitemap unaided"] From 4052ecbb8f8bf4afa47e2e844ebe8024ac42cdde Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 07:47:48 -0700 Subject: [PATCH 5/7] Print the robots summary after its last assertion, not before Two findings, both correct. The advertising line printed before the sitemap-existence check ran, so the one failing case read as a pass on the line immediately above its own failure. Now every branch prints exactly once, after deciding: missing-sitemap reports "1 advertised sitemap(s) not built" and only a clean run prints what it advertises. The TODO paragraph describing the cutover loss still said "since this site emits none" in the present tense, one bullet below the entry saying the file is now built. It reads as history now, which is what it is. Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 2 +- checks/check-url-parity.py | 13 ++++++++----- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/TODO.md b/TODO.md index 2ad6e2a..6795791 100644 --- a/TODO.md +++ b/TODO.md @@ -35,7 +35,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. - **`robots.txt` is decided and built, 2026-08-08, and what remains is that production has not been redeployed since.** The site emits one now, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. The record below is kept because the reasoning is what the next decision about crawl directives will need, and because production still answers 404 until a deploy carries this. - **The first deploy did not fix the 404, and that is what turned this from a gap into a decision.** The VPS agent raised it in §22.10 and both halves were measured rather than assumed: the site emitted no `robots.txt` at all, because `hugo.yaml` set no `enableRobotsTXT`, so the 404 survived the deploy and `X-Robots-Tag` was the only control, while `sitemap.xml` **was** emitted and became fetchable on the interim name at that same deploy. A crawler got a full sitemap and no robots file. `enableRobotsTXT` is now set, so this describes the release production is still serving rather than the current build. - - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Since this site emits none, M7b is not a return to a previous state, it is a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer goes with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **That is what was chosen**, out of three options: preserve today's behavior, write what this site actually wants, or keep emitting nothing and accept the loss. The sitemap URL is derived from the built `baseURL` rather than typed, which is what makes the choice survive the cutover without a second edit. + - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Because this site emitted none, M7b would not have been a return to a previous state, it would have been a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer would have gone with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **That is what was chosen**, out of three options: preserve today's behavior, write what this site actually wants, or keep emitting nothing and accept the loss. The sitemap URL is derived from the built `baseURL` rather than typed, which is what makes the choice survive the cutover without a second edit. - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel. - **`/robots.txt/`, with a trailing slash, now redirects to the real file** rather than to the home page, in the same change, since the two are only correct together. The fix is in `build-redirects.py` rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. `/osd.xml/` stays pointed at the home page: it was the old platform's OpenSearch description and this site emits no such file. - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index b78b083..e250f7a 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -133,13 +133,16 @@ def check_robots(public): print(f"robots : built, {len(wrong)} Sitemap line(s) naming another origin") return [f"{u} (this build's origin is {origin})" for u in wrong] + # The summary is printed after the last assertion rather than before it, or the missing-sitemap + # case reads as a pass on the line above its own failure. Every branch here prints exactly once. + unbuilt = [u for u in advertised if not (public / u[len(origin) + 1 :]).is_file()] + if unbuilt: + print(f"robots : built, {len(unbuilt)} advertised sitemap(s) not built") + return [f"{u} (advertised, but {u[len(origin) + 1:]} was not built)" for u in unbuilt] + # Named rather than counted, since there is one today and the line is the thing being checked. print(f"robots : built, advertising {advertised[0]}") - return [ - f"{u} (advertised, but {u[len(origin) + 1:]} was not built)" - for u in advertised - if not (public / u[len(origin) + 1 :]).is_file() - ] + return [] def site_origin(public): From 12c4ae2b77bca396ceeb89cdafcc4e574db8ddc8 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 07:51:29 -0700 Subject: [PATCH 6/7] Describe the missing robots.txt rather than asserting why it is missing The message named enableRobotsTXT as the cause. A partial build, a deleted file, and the wrong output directory reach the identical state, so asserting one sends a reader to verify a setting that is already correct. It now names the path it looked for, offers the likely cause as likely, and says the alternatives are indistinguishable from here, which is the same shape the orphan messages use for the same reason. Co-Authored-By: Claude Opus 5 (1M context) --- checks/check-url-parity.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index e250f7a..70b7b2d 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -115,8 +115,14 @@ def check_robots(public): """ robots = public / "robots.txt" if not robots.is_file(): + # Naming one cause as the cause sends a reader to check a setting that is already correct. + # enableRobotsTXT is the likely one and a partial build or the wrong output directory reach + # the same state, which is the same reason the orphan messages name both of their causes. print("robots : missing") - return ["robots.txt was not built - enableRobotsTXT is off in hugo.yaml"] + return [ + f"{robots} does not exist - likely enableRobotsTXT is unset in hugo.yaml, " + "though a partial build or the wrong output directory look identical here" + ] origin = site_origin(public) # errors="replace" rather than strict, or invalid UTF-8 raises out of the whole parity run and a From 0b4e629cb3b10382729df72b87a5d51c2ef8fba7 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Sat, 8 Aug 2026 07:55:39 -0700 Subject: [PATCH 7/7] Pluralize the failure noun by count, which fixes the gallery too "FAIL robots: 1 problems" was the report. The noun was a bare string and a count is always printed beside it, so every label with a plural noun had the same defect: the pre-existing gallery one prints "1 stray nodes" on a single stray node, which is the shape that made the new noun wrong in the same way. Now a (singular, plural) pair chosen by count. The default stays "missing" and needs no pair, being count-neutral already. Verified across both labels at 1 and at many, and the clean run is unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- checks/check-url-parity.py | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index 70b7b2d..86a24e5 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -41,8 +41,15 @@ # Every check above returns a list, and the shared summary called all of them "missing". That is # what a URL that did not build is, and it is not what a stray node inside a gallery is: those are # present, which is the whole complaint. The default stays "missing" so a check added later reads -# the way the older ones do unless it says otherwise. -FAILURE_NOUN = {"gallery": "stray nodes", "robots": "problems"} +# the way the older ones do unless it says otherwise, and it needs no pair because it is already +# count-neutral. A count is always printed beside the noun, so the pair is (singular, plural) and +# "1 stray nodes" was the reason for making it a pair rather than a string. +FAILURE_NOUN = {"gallery": ("stray node", "stray nodes"), "robots": ("problem", "problems")} + + +def failure_noun(label, count): + forms = FAILURE_NOUN.get(label, ("missing", "missing")) + return forms[0] if count == 1 else forms[1] def load(name): @@ -377,7 +384,7 @@ def main(argv): print() for label, found in failures: - print(f"FAIL {label}: {len(found)} {FAILURE_NOUN.get(label, 'missing')}") + print(f"FAIL {label}: {len(found)} {failure_noun(label, len(found))}") for item in found[:20]: print(f" {item}") if len(found) > 20: