Fetch media in the live check, so a lost image is caught somewhere

[ptr727]

The build gate proves the media set, against files on disk. Nothing proved those files reached the server or that the server can read them: the live check requested pages and redirects and never one image, so a media tree lost between a passing build and the server was caught by neither gate. On a site whose value is eighteen years of posts with images in them, that was the gap worth closing.

This also unblocks git-restore-mtime, which was deliberately held behind it.

What the list is, and why it is short

checks/golden-media-live.txt is a handful rather than exhaustive, because the set is already proven and this is a delivery check. The entries cover both trees, which arrive by different routes and where a partial transfer is unlikely to land evenly, plus the legacy form:

Entry shape	Proves
/media/ paths	the imported uploads tree arrives and is served
/external/ paths	the tree of media localized from other hosts arrives too
/wp-content/uploads/ paths	the @uploads rule still lands on the image, which nothing else exercised against a running server

Every entry was verified with a live request before being added, like every other list here.

Three assertions, because each catches a different loss

Failure	Answers
file missing from the transfer	404
file whose mode went wrong	403
file truncated to nothing	200, zero bytes
server returning an error page for a missing asset	200, text/html

The 403 case is why this is not theoretical. A hard-linked file carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server. A build-time is_file() on the runner cannot see it, and neither could a check that never requests an image.

check_media follows one hop by hand rather than passing -L to curl, because -L would carry the auth-gate credential to wherever the rule points. Same origin boundary as check_redirect, for the same reason.

Verification

Against production, PASS - 1253 URLs honored, with the media step running.

Each failure shape was reproduced rather than assumed, and each names the list entry rather than the resolved URL:

media /media/2016/01/does-not-exist.jpg expected 200, got 404
media /wp-content/uploads/2016/01/gone.jpg expected 200, got 404     <- hop followed, destination measured
media /about/ answered 200 as text/html; charset=utf-8, expected an image
media <zero-byte file> answered 200 with an empty body               <- against a throwaway server

The three legacy entries passing is itself the proof that the hop is followed: unfollowed they would have failed as 301.

One judgment worth flagging

The legacy entries went in this list rather than redirect-urls.txt, where they would also have worked. Adding them there moves a count stated in six documents and in the published migration post (328 + 917 = 1,245), which is disproportionate to three test URLs. The documented counts are unchanged: render 328, redirect 917.

Gates

shellcheck and shfmt -d clean on all four scripts, markdownlint 18 files clean, editorconfig clean, check-env-docs.py 28/28, and the build gate unaffected. The hub's prose_lint.py is at its exact baseline for both files touched.

[Copilot]

Pull request overview

Adds a small, targeted set of live media URL checks to the existing “live URL contract” gate so that missing/unreadable/truncated images (or soft-404 HTML responses) are caught after deployment, not just at build time.

Changes:

• Introduces checks/golden-media-live.txt as a short live-request list covering /media/, /external/, and legacy /wp-content/uploads/ URLs.
• Extends checks/check-live-urls.sh with a check_media step that asserts status, non-empty body, and image content type (with a redirect hop followed safely).
• Documents the rationale and closes the corresponding TODO entry.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
TODO.md	Records the new live media check as completed and unblocks the mtime restoration follow-up item.
checks/README.md	Documents why a second (live) media list exists and what it asserts.
checks/golden-media-live.txt	Adds the curated live media URL list used by the live gate.
checks/check-live-urls.sh	Implements the new parallelized live media verification and includes it in totals/floors.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

checks/check-live-urls.sh:104

• check_media parses curl output with read -r code len type <<<"$out" while the content_type field can legally contain spaces (e.g. text/html; charset=utf-8). With the current space-separated -w format, type will be truncated at the first space, which reduces diagnostic quality and can diverge from the expected failure messages described in the PR (e.g. it would report text/html; instead of text/html; charset=utf-8). Use a delimiter that cannot appear in the content type (tab) and set IFS accordingly when reading.

	out=$(curl -s -o /dev/null \
		-w '%{http_code} %{size_download} %{content_type}\n' \
		--max-time 30 "${target_auth[@]}" "$target") || rc=$?
	read -r code len type <<<"$out"
	if [ "$rc" -ne 0 ] || [ "${code:-000}" = "000" ]; then

[ptr727]

One suppressed finding, and this is the first one today I am declining. Measured rather than argued.

---

checks/check-live-urls.sh:104 — check_media parses curl output with read -r code len type <<<"$out" while the content_type field can legally contain spaces (e.g. text/html; charset=utf-8). With the current space-separated -w format, type will be truncated at the first space … it would report text/html; instead of text/html; charset=utf-8.

The premise about content types containing spaces is right. The consequence is not. read with N variables assigns the entire remainder of the line to the Nth, spaces included, so type receives the full value rather than being truncated:

$ out='200 1234 text/html; charset=utf-8'
$ read -r code len type <<<"$out"; printf 'type=[%s]\n' "$type"
type=[text/html; charset=utf-8]

And against a real response rather than a literal:

$ o=$(curl -s -o /dev/null -w '%{http_code} %{size_download} %{content_type}\n' https://<site>/about/)
$ read -r c l t <<<"$o"; printf 'type=[%s]\n' "$t"
type=[text/html; charset=utf-8]

The failure message quoted in the PR description was captured from an actual run against production, not written by hand, which is the other reason it already shows the full type.

Truncation would happen if the field order put content_type anywhere but last, or if a fourth -w field were appended after it. That is a real hazard for a later edit rather than a current defect, so rather than change the parsing I would rather the constraint be visible at the place someone would break it. So: declining the tab-delimiter change, and in 66c49b9 the constraint is now written on the format string itself, where someone adding a fourth field would read it.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

checks/README.md:72

• The README’s quick “check the counts” command doesn’t include the newly added golden-media-live.txt, so following it won’t reflect all check lists currently in use.

wc -l checks/golden-urls.txt checks/redirect-urls.txt checks/golden-media-legacy.txt

[ptr727]

checks/README.md:72 — The README's quick "check the counts" command doesn't include the newly added golden-media-live.txt, so following it won't reflect all check lists currently in use.

Correct, and it is the one kind of staleness this file least affords. That command exists so a reader checks a count rather than trusting a written one, and it had itself become a written thing that drifted. Fixed in 3d6eb14, and I confirmed it now names every list check-live-urls.sh reads rather than just adding the obvious one.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

checks/check-live-urls.sh:23

• The FLOOR/truncation guard can be bypassed if grep fails to read a list file (missing/unreadable). Because the script does not use set -e, a failing n=$(grep ...) can yield an empty n, and the numeric test may error and evaluate false, allowing the script to continue and potentially run with an empty input set.

for list in golden-urls.txt redirect-urls.txt golden-media-live.txt; do
	n=$(grep -c . "$CHECKS/$list")
	if [ "$n" -lt "${FLOOR[$list]}" ]; then
		echo "FAIL $list: $n URLs, expected at least ${FLOOR[$list]} - the list has been truncated" >&2
		exit 1

checks/check-live-urls.sh:93

• When following a 301/308, curl -w '%{redirect_url}' can return an empty string (e.g., missing/invalid Location header) or fail, which currently leads to a subsequent curl "$target" with an empty/invalid URL and a misleading "transport error" message. Validate that a redirect target was actually obtained and fail with a clear message if not.

	301 | 308)
		target=$(curl -s -o /dev/null -w '%{redirect_url}' --max-time 30 "${auth[@]}" "$target")
		# Same origin boundary as check_redirect, and for the same reason: a rule that one
		# day points off-site must not mail the token there. A bare prefix would also accept
		# a lookalike host registered as an attacker's subdomain.

[ptr727]

Both real, both fixed in 5b92b46. The first is the more serious finding on this branch.

---

checks/check-live-urls.sh:23 — The FLOOR/truncation guard can be bypassed if grep fails to read a list file … a failing n=$(grep ...) can yield an empty n, and the numeric test may error and evaluate false, allowing the script to continue.

Correct, and reproduced exactly as described:

n=[]
/bin/bash: line 15: [: : integer expression expected
GUARD DID NOT FIRE - script would continue
outer rc=0

A guard that fails open is worse than no guard, and this one sits above every assertion in the file: the run would continue with an empty input set and exit 0, having checked nothing. That is the exact shape this repository has a standing rule about — a query that matched nothing reading as a clean result — so it is a good catch against code that was written by someone who knows the rule.

Readability is now asserted before the count, and the count is validated as a number before it is compared. Verified all three paths:

FAIL golden-media-live.txt: not readable at …/checks/golden-media-live.txt     exit 1
FAIL golden-media-live.txt: 2 URLs, expected at least 8 - the list has been truncated   exit 1
PASS - 1253 URLs honored

Worth noting the scope: this guard predates the branch and covered two lists before it covered three. The hole was pre-existing and this change is where it surfaced.

---

checks/check-live-urls.sh:93 — When following a 301/308, curl -w '%{redirect_url}' can return an empty string … which currently leads to a subsequent curl "$target" with an empty/invalid URL and a misleading "transport error" message.

Correct. It still failed closed, but it named the wrong problem, and a message that misdirects costs more than a missing one when someone is reading a failed gate at speed. An empty target is now reported as answered 301 with no usable Location.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.