Emit robots.txt, and gate the base URL it is derived from

[ptr727]

Replaces #59, which GitHub closed when its base branch was deleted on the #58 merge. Same seven commits, rebased onto develop with #58's two commits dropped, so the diff is only this change: 6 files.

Two things carried over from #59 that matter.

#59 never ran CI, and I reported it as green. test-pull-request.yml triggers on pull_request: branches: [main, develop], and #59 was opened against a feature branch to stack it, so the validation workflow never fired. Its checks=2/2 was two skipped jobs. This PR targets develop, so the gate runs for real.

Six review rounds on #59 found six things, all accepted, none declined. Every fix is in the commits here, and the round-by-round exchange is on #59 for the record:

Round	Finding
1–2	Four places claimed this check makes a wrong HUGO_BASEURL visible. It cannot — both sides of the comparison derive from baseURL — withdrawn
3	read_text at strict UTF-8 would raise UnicodeDecodeError and take down the whole parity run, in the one gate written to report a bad robots.txt — fixed
4	The summary printed robots : built, advertising … above its own failure — fixed
5	The missing-file message asserted enableRobotsTXT as the cause where three causes are indistinguishable — fixed
6	FAIL robots: 1 problems, which the pre-existing gallery label shared — fixed for both

coverage=PARTIAL at 5 of 6 files held in every one of those rounds, which the runbook says a re-request does not cure. Worth watching whether it clears here now that the base is develop.

---

The old platform serves a robots.txt today and this site emitted none, so the cutover was not a return to a previous state. It was a move from having crawl directives to having none, on a site that has had them for years, and the sitemap pointer went with them. Hugo emits the file only when enableRobotsTXT is set, which is why /robots.txt answered 404 through the first production deploy.

The Sitemap: line is the load-bearing part, not the rules

Across the interim hostname's first full day, every request for sitemap.xml and feed.xml came from curl — the deploy gate's or the host side's — and none from a crawler, while /robots.txt was requested nine times and 404'd every time, five of those from real agents on a name with no inbound links. A crawler is told where a sitemap is rather than guessing it, and the only thing telling them is the file the cutover deletes.

Every Disallow the old platform serves names a WordPress path this site does not have, so the content that preserves today's behaviour is User-agent: *, no Disallow, and the sitemap:

User-agent: *
Disallow:
Sitemap: https://blog.insanegenius.com/sitemap.xml

The theme's template already derives that URL from the built baseURL, so setting the flag is the whole change and there is no second edit to remember at M7b. Verified across two base URLs rather than reasoned about: a mirror build advertises https://blog.home.insanegenius.net/sitemap.xml and a production build advertises the .com one, from the same bytes.

What check_robots gates, and what it does not

Four failures, each demonstrated failing before the check was trusted:

Case	Reported
robots.txt absent	<public>/robots.txt does not exist - likely enableRobotsTXT is unset in hugo.yaml, though a partial build or the wrong output directory look identical here
no Sitemap: line	robots.txt carries no Sitemap: line - a crawler will not find the sitemap unaided
line naming another origin	https://blog.insanegenius.net/sitemap.xml (this build's origin is https://blog.insanegenius.com)
line advertising a sitemap not built	https://blog.insanegenius.com/news-sitemap.xml (advertised, but news-sitemap.xml was not built)

Each exits 1, and each prints exactly one summary line, after deciding rather than before.

It does not detect a wrong HUGO_BASEURL. The advertised origin and the canonical origin it is compared against both derive from baseURL, so they agree whenever the build is coherent, including when baseURL was wrong for the target environment. Nothing inside the artifact can see that, so the gap stays on the side that knows which host it is serving: the VPS agent reads the origin out of the deployed sitemap.xml, og:url and feed.xml, which is how the first production deploy was cleared.

What the origin comparison is genuinely for is an origin written rather than derived: a committed static/robots.txt shadowing the template, with the old platform's .com sitemap line pasted into it, is the mistake it catches.

/robots.txt/ reaches the file rather than the home page

/robots.txt/, with a trailing slash, is a URL the old platform served and is in the redirect contract. It resolved through slugs.map as an unresolvable attachment slug, which sends a visitor to the home page — right for a slug nothing claims, wrong for a file that now exists.

Fixed in build-redirects.py rather than in the generated map, because the map is rewritten from the capture and a hand edit does not survive the next regeneration. Confirmed by regenerating first and diffing: the committed maps were byte-identical to a fresh run, so the map diff here is exactly the one line. /osd.xml/ stays pointed at the home page deliberately, being an OpenSearch description this site does not emit.

Verification

This touches deploy/maps/ and hugo.yaml, so the local mirror gate applies rather than CI's green being sufficient:

PASS - the built site honors the URL contract
robots : built, advertising https://blog.home.insanegenius.net/sitemap.xml
==> installing release 20260808-142254
==> served by mirror-production
PASS - 1245 URLs honored

/robots.txt/  301 -> https://blog.home.insanegenius.net/robots.txt
/robots.txt   200 text/plain; charset=utf-8
/osd.xml/     301 -> https://blog.home.insanegenius.net/          (unchanged)

Re-verified after the rebase onto develop: build clean under --panicOnWarning, parity gate PASS.

Production still answers 404 for the file until a deploy carries this, which is recorded in TODO.md rather than assumed.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR ensures the site emits a robots.txt (including a sitemap pointer derived from the built baseURL) and adds a deploy gate to verify that the generated robots.txt is present, internally consistent with the build’s origin, and only advertises sitemaps that were actually built. It also updates redirect generation so /robots.txt/ lands on the real file.

Changes:

• Enable Hugo enableRobotsTXT so /robots.txt is emitted and includes a Sitemap: line derived from the built baseURL.
• Add a check_robots() gate to check-url-parity.py, and improve failure summary wording to handle singular/plural properly.
• Update redirect generation and the committed slug map so /robots.txt/ redirects to /robots.txt, plus documentation updates capturing the decision and its limits.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
TODO.md	Updates the operational record/decision log for robots.txt, including the gate’s scope and limitations.
hugo.yaml	Enables enableRobotsTXT: true to emit robots.txt.
deploy/maps/slugs.map	Redirects /robots.txt/ to /robots.txt instead of /.
checks/README.md	Documents what the robots gate checks and the hard limit it cannot detect (wrong external base URL).
checks/check-url-parity.py	Adds check_robots() and improves failure noun handling (singular/plural), wiring the new gate into the parity run.
checks/build-redirects.py	Special-cases /robots.txt/ as a well-known root file redirect during map generation.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.