diff --git a/TODO.md b/TODO.md index a8cbb5b..6795791 100644 --- a/TODO.md +++ b/TODO.md @@ -33,11 +33,12 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. -- **Decide what `robots.txt` says, now before the cutover rather than before the first deploy.** Production answers 404 for it, so `X-Robots-Tag` is the only thing keeping the interim hostname out of an index. Crawlers are already asking: OAI-SearchBot, ClaudeBot, GPTBot, and Scrapy each arrived within three hours of the certificate being issued, at a hostname with no inbound links. - - **The deploy did not fix the 404, which is the part that reads the wrong way round, and is now measured rather than predicted.** The VPS agent raised this in §22.10 and asked two questions of this side. Both are now measured rather than assumed: **this site emits no `robots.txt` at all**, because `hugo.yaml` sets no `enableRobotsTXT`, so the 404 survives the deploy and `X-Robots-Tag` stays the only control. **`sitemap.xml` is emitted** and becomes fetchable on the interim name at that same deploy, listing whatever `HUGO_BASEURL` was built with, which is the second reason that variable mattered. Read together: after the rehearsal deploy a crawler gets a full sitemap and no robots file. - - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Since this site emits none, M7b is not a return to a previous state, it is a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer goes with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the `.com` sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **Deliberately undecided**, and the three options are to preserve today's behavior, to write what this site actually wants, or to keep emitting nothing and accept the loss. +- **`robots.txt` is decided and built, 2026-08-08, and what remains is that production has not been redeployed since.** The site emits one now, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. The record below is kept because the reasoning is what the next decision about crawl directives will need, and because production still answers 404 until a deploy carries this. + - **The first deploy did not fix the 404, and that is what turned this from a gap into a decision.** The VPS agent raised it in §22.10 and both halves were measured rather than assumed: the site emitted no `robots.txt` at all, because `hugo.yaml` set no `enableRobotsTXT`, so the 404 survived the deploy and `X-Robots-Tag` was the only control, while `sitemap.xml` **was** emitted and became fetchable on the interim name at that same deploy. A crawler got a full sitemap and no robots file. `enableRobotsTXT` is now set, so this describes the release production is still serving rather than the current build. + - **At the cutover this stops being a gap and becomes a loss, which is the half neither side had checked.** The live `.com` blog **serves a `robots.txt` today, carrying a `Sitemap:` line**. Because this site emitted none, M7b would not have been a return to a previous state, it would have been a move from having crawl directives to having none on a site that has had them for years, and the sitemap pointer would have gone with them. The VPS agent measured this from the outside in §23.3, will not put a file in this repository's bundle, and has made it a decision that blocks step 1 of the M7b checklist rather than one discovered after it. The minimum that preserves today's behavior is `User-agent: *`, no `Disallow`, and the sitemap line, since every `Disallow` the old platform serves names a WordPress path this site does not have. **That is what was chosen**, out of three options: preserve today's behavior, write what this site actually wants, or keep emitting nothing and accept the loss. The sitemap URL is derived from the built `baseURL` rather than typed, which is what makes the choice survive the cutover without a second edit. - **The log reframes the decision, and it is the `Sitemap:` line that carries it rather than any rule.** Across the interim hostname's first full day, `/robots.txt` was requested nine times and answered 404 every time, five of those from real agents on a hostname with no inbound links. **No crawler fetched `sitemap.xml` or `feed.xml` once**: every request to either came from `curl`, the deploy gate's or the host side's. Crawlers do not guess a sitemap's location, they are told it, and the only thing telling them today is the `robots.txt` the old platform serves, which is the file the cutover deletes. So the question is not whether to have crawl directives, it is whether the sitemap stays advertised at all. Measured on the host side in its §26.4 and recorded here because the decision outlives that channel. - - **`/robots.txt/`, with a trailing slash, is an entry in `slugs.map` pointing at `/`**, noted under "Open decisions" below. If a real file lands, that entry points at it instead, and the change ships in the same release rather than after it. + - **`/robots.txt/`, with a trailing slash, now redirects to the real file** rather than to the home page, in the same change, since the two are only correct together. The fix is in `build-redirects.py` rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. `/osd.xml/` stays pointed at the home page: it was the old platform's OpenSearch description and this site emits no such file. + - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. - **Nothing checks that media survived the trip to the server.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. - **Restore file mtimes in CI so `--link-dest` links, and do it after the media check rather than before.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. @@ -72,7 +73,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no - **The same question covers the migration toolchain in the capture directory**, which is fourteen scripts: the `wp2hugo` run, the content restructure and clean passes, external-media localization, the crawl and mirror, the golden-URL build, and the media inventory. Some are worth keeping only if generalized, and some are cheaper to rewrite than to maintain, so this is a per-script call rather than one decision. - **One of them is already three copies with two of them stale**, which is the concrete version of this risk rather than a hypothetical one. `build-redirects.py` exists at the capture root, again under the capture's own `checks/`, and here at [`checks/build-redirects.py`](./checks/build-redirects.py). The two capture copies are identical to each other at 115 lines; the copy in this repository is the maintained one at 225. Nothing detects that, because the capture is not a git repository and is read-only in normal use. - **What `robots.txt` says, which is undecided and is the last non-mechanical item before M7b.** Recorded under "Next" above, where it blocks the cutover. -- `/robots.txt/` and `/osd.xml/` currently sit in `slugs.map` pointing at `/`. The first would be better pointing at the real `/robots.txt`. +- `/osd.xml/` sits in `slugs.map` pointing at `/`, and stays there unless this site ever emits an OpenSearch description. `/robots.txt/` was the other half and is resolved, above. - Content is capped at a fixed 720px on every screen, because PaperMod's width is four CSS variables with no responsive term and no Hugo parameter. The prose measure is right and should stay; images and galleries inheriting the same cap is the part that costs something on a wide display. The knobs, the override location, and the `--gap` trap are documented under "Customization points" in [`themes/README.md`](./themes/README.md). ## Deliberate deviations from the fleet baseline diff --git a/checks/README.md b/checks/README.md index 83bc682..89d866a 100644 --- a/checks/README.md +++ b/checks/README.md @@ -84,6 +84,18 @@ A count is all the check can observe, and two causes reach each direction: it ri **Both directions read absolute references as well as relative ones.** Hugo writes an absolute URL wherever a template resolves one against the base, which the entry-cover image on every list page does. Reading only rooted paths made those files look linked from nowhere while they were being displayed, and left a broken one unchecked in the other direction. The origin is read from the home page's canonical link rather than assumed, since staging and production build with different base URLs and a hardcoded host would check one environment's output against another's. No canonical link is a hard failure, because a guessed origin inflates the orphan count by exactly the pages that use one. +## The robots check, and the one thing no gate here can do + +Hugo emits no `robots.txt` unless `enableRobotsTXT` is set, so this site served none until the flag was turned on, and the old platform serves one. The `Sitemap:` line is load-bearing in a way the crawl rules are not: across the interim hostname's first full day of traffic, every request for `sitemap.xml` came from `curl` and none from a crawler, because a crawler is told where a sitemap is rather than guessing it. Four failures are gated — the file absent, no `Sitemap:` line, a line naming another origin, and a line advertising a sitemap that was not built — and all four were demonstrated failing before the check was trusted. + +**The `Sitemap:` line is the one place this gate compares an origin rather than joining one.** Every list is path-only and `check-live-urls.sh` joins whatever base URL it is handed, which is deliberate and is what lets one contract cover four environments. Other absolute URLs are read here, the home page's canonical link and the absolute asset references described above among them, but they are read to resolve a reference rather than to check one host against another. + +**What that comparison proves is internal consistency, and it is worth being exact about the limit.** The advertised origin must match the one read from the home page's canonical link, and both are derived from the same `baseURL`, so they agree whenever the build is coherent — including when `baseURL` held the wrong value for the environment being deployed to. **Nothing inside the artifact can detect that**, which is why the check belongs on the side that knows which host it is serving: the VPS side reads the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml` and reports the counts either way. A build baked with the wrong host still passes all 1,245 URLs here, and it passes this too. + +What the comparison does catch is an origin that was **written rather than derived** — a committed `static/robots.txt` shadowing the template is the way that happens, and pasting the old platform's `.com` sitemap line into one is the specific mistake it would catch — along with a sitemap advertised but never built. + +`/robots.txt/`, with a trailing slash, is a URL the old platform served and is in the redirect contract. It resolves through `slugs.map` like any other one-segment legacy URL, and the generator special-cases it to the real file rather than to the home page. `/osd.xml/` stays pointed at the home page deliberately: it was the old platform's OpenSearch description and this site emits no such file. + ## The gallery check, which no direction above can reach Every check above reasons about a URL: whether it renders, whether it resolves, whether anything points at it. Content misplaced **inside** a gallery satisfies all of that. The file exists, the reference resolves, and something links it, so the media surface is green in both directions while the page is laid out wrong. The defect is one of structure, which is why three variants of it survived the conversion and every gate since. diff --git a/checks/build-redirects.py b/checks/build-redirects.py index 414ef9f..3d94171 100755 --- a/checks/build-redirects.py +++ b/checks/build-redirects.py @@ -19,6 +19,15 @@ # The WordPress importer registers the full slug, and both answer, so both are mapped. BLOGGER_SLUG_LIMIT = 40 +# A one-segment URL naming a file the site serves at the root, rather than a page slug. +# The resolver below reads these as unresolvable attachment slugs and sends them to the home +# page, which is the right answer for a slug nothing claims and the wrong one for a file that +# exists: /robots.txt/ should reach /robots.txt. Named here rather than hand-edited into the +# generated map, because the map is rewritten from the capture and a hand edit does not survive. +# /osd.xml/ stays out deliberately. It was the old platform's OpenSearch description and this +# site emits no such file, so the home page remains the honest destination for it. +WELL_KNOWN = {"/robots.txt/": "/robots.txt"} + def text(el, path): node = el.find(path, NS) @@ -179,7 +188,9 @@ def main(argv): resolved, via_parent, via_media, orphan = [], 0, 0, [] for u in needed: slug = u.strip("/") - if slug in by_slug: + if u in WELL_KNOWN: + resolved.append((u, WELL_KNOWN[u])) + elif slug in by_slug: resolved.append((u, by_slug[slug])) via_parent += 1 elif slug.lower() in file_to_post: diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index 2d017fa..86a24e5 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -41,8 +41,15 @@ # Every check above returns a list, and the shared summary called all of them "missing". That is # what a URL that did not build is, and it is not what a stray node inside a gallery is: those are # present, which is the whole complaint. The default stays "missing" so a check added later reads -# the way the older ones do unless it says otherwise. -FAILURE_NOUN = {"gallery": "stray nodes"} +# the way the older ones do unless it says otherwise, and it needs no pair because it is already +# count-neutral. A count is always printed beside the noun, so the pair is (singular, plural) and +# "1 stray nodes" was the reason for making it a pair rather than a string. +FAILURE_NOUN = {"gallery": ("stray node", "stray nodes"), "robots": ("problem", "problems")} + + +def failure_noun(label, count): + forms = FAILURE_NOUN.get(label, ("missing", "missing")) + return forms[0] if count == 1 else forms[1] def load(name): @@ -92,6 +99,65 @@ def check_media(public): return missing +def check_robots(public): + """Check that robots.txt exists and advertises this build's own sitemap. + + Four failures, one gate: the file absent, no Sitemap: line, a line naming another origin, and a + line advertising a sitemap that was not built. Hugo emits no robots.txt unless enableRobotsTXT + is set, which is why this site served none until that flag was turned on, and the file's only + load-bearing line points at the sitemap a crawler is otherwise unlikely to find, being told + rather than guessing. + + Every contract list is path-only and check-live-urls.sh joins whatever base URL it is handed, so + this is the only place the gate *compares* an origin rather than joining one. Other absolute + URLs are read here - the home page's canonical link, and absolute asset references - but they + are read to resolve a reference rather than to check a host against another. + + What the comparison proves is internal consistency: the advertised origin matches the one the + canonical link carries. Both come from baseURL, so this cannot tell that baseURL was the wrong + value for the environment being deployed to - nothing in the artifact can, which is why that + check belongs on the side that knows which host it is serving. It does catch an origin that was + written rather than derived, a static robots.txt shadowing the template being the way that + happens, and it catches a sitemap advertised but not built. + """ + robots = public / "robots.txt" + if not robots.is_file(): + # Naming one cause as the cause sends a reader to check a setting that is already correct. + # enableRobotsTXT is the likely one and a partial build or the wrong output directory reach + # the same state, which is the same reason the orphan messages name both of their causes. + print("robots : missing") + return [ + f"{robots} does not exist - likely enableRobotsTXT is unset in hugo.yaml, " + "though a partial build or the wrong output directory look identical here" + ] + + origin = site_origin(public) + # errors="replace" rather than strict, or invalid UTF-8 raises out of the whole parity run and a + # gate that exists to report a bad robots.txt stack-traces on one instead. A committed + # static/robots.txt is the file most likely to carry it, and it is the case this check is for. + text = robots.read_text(encoding="utf-8", errors="replace") + advertised = re.findall(r"(?mi)^\s*Sitemap:\s*(\S+)\s*$", text) + if not advertised: + print("robots : built, no Sitemap line") + return ["robots.txt carries no Sitemap: line - a crawler will not find the sitemap unaided"] + + wrong = [u for u in advertised if not u.startswith(origin + "/")] + if wrong: + print(f"robots : built, {len(wrong)} Sitemap line(s) naming another origin") + return [f"{u} (this build's origin is {origin})" for u in wrong] + + # The summary is printed after the last assertion rather than before it, or the missing-sitemap + # case reads as a pass on the line above its own failure. Every branch here prints exactly once. + unbuilt = [u for u in advertised if not (public / u[len(origin) + 1 :]).is_file()] + if unbuilt: + print(f"robots : built, {len(unbuilt)} advertised sitemap(s) not built") + return [f"{u} (advertised, but {u[len(origin) + 1:]} was not built)" for u in unbuilt] + + # Named rather than counted, since there is one today and the line is the thing being checked. + print(f"robots : built, advertising {advertised[0]}") + return [] + + def site_origin(public): """The site's own scheme and host, read from the artifact rather than assumed. @@ -307,6 +373,7 @@ def main(argv): ("assets", check_assets(public, refs)), ("orphans", check_orphans(public, refs)), ("gallery", check_galleries(public)), + ("robots", check_robots(public)), ): if found: failures.append((label, found)) @@ -317,7 +384,7 @@ def main(argv): print() for label, found in failures: - print(f"FAIL {label}: {len(found)} {FAILURE_NOUN.get(label, 'missing')}") + print(f"FAIL {label}: {len(found)} {failure_noun(label, len(found))}") for item in found[:20]: print(f" {item}") if len(found) > 20: diff --git a/deploy/maps/slugs.map b/deploy/maps/slugs.map index d27247c..2eb63c0 100644 --- a/deploy/maps/slugs.map +++ b/deploy/maps/slugs.map @@ -100,7 +100,7 @@ /jellyfish-lighting-ports-500x214-1/ / /osd.xml/ / /packet/ /2020/09/05/abandoning-greeneye-for-sense/ -/robots.txt/ / +/robots.txt/ /robots.txt /sce_smartmeter/ /2020/09/05/abandoning-greeneye-for-sense/ /ted-dashboard/ /2020/09/05/abandoning-greeneye-for-sense/ /ted-graphing/ /2020/09/05/abandoning-greeneye-for-sense/ diff --git a/hugo.yaml b/hugo.yaml index 2ec8d0f..ded53f7 100644 --- a/hugo.yaml +++ b/hugo.yaml @@ -27,6 +27,15 @@ refLinksErrorLevel: ERROR # The deploy rebuilds the site rather than receiving it as an artifact, so the build stays reproducible. enableGitInfo: false +# Off by default in Hugo, which is why this site answered 404 for a file the old platform served. +# The Sitemap: line is the load-bearing part: across the interim hostname's first full day of +# traffic, every request for sitemap.xml came from curl and none from a crawler, because a crawler +# is told where a sitemap is rather than guessing, and the only thing telling them today is the +# robots.txt the cutover deletes. +# The theme's template derives that URL from the built baseURL, so it names .net during the +# rehearsal and .com afterwards with no second edit to remember at the cutover. +enableRobotsTXT: true + params: description: My discoveries, frustrations, and delights defaultTheme: auto