Announce synthetic traffic with X-Blog-Check on every live-check request

[ptr727]

Implements the header the VPS agent proposed in their §31, with one refinement. Their host already captures the field and their ci/smoke.sh already sends vps/smoke; this is the other half.

Why a coincidence was holding this up

97.7% of the traffic reaching production is synthetic, and the only thing separating it from real visitors was accidental. Measured on their side:

Traffic	User-Agent	Client
hand runs from this host	curl/8.14.1	stable
the CI deploy gate	curl/8.5.0	three Azure addresses on 08-08 alone, never repeating
their smoke.sh	curl/8.5.0	stable

The CI runner's curl and the VPS host's curl are byte-identical, so UA cannot separate our gate from their probe — only client address can, and the CI half of that rotates every run. One runner-image bump merges the two and nothing reports it.

The design, and the one refinement

Agreed as proposed: one request header carrying provenance rather than a boolean, so "which run produced this 404" is a single query.

X-Blog-Check: <source>/<id>

The refinement: the run attempt is part of the id. A re-run of a failed workflow keeps the same GITHUB_RUN_ID and takes a new GITHUB_RUN_ATTEMPT, so the id alone merges a retry into the run it was retrying — exactly the case someone reads the log to understand. So github/<run-id>-<attempt> rather than github/<run-id>.

Derived rather than configured, so a hand run is never accidentally untagged:

Context	Value
GitHub Actions	github/31291656581-2
anywhere else	proxmox/manual
CHECK_TAG set	verbatim, e.g. proxmox/media-dev

Two config files, which is the part not to collapse later

The token is sent only to the origin it belongs to — a redirect that one day points off-site must not mail the credential there. The tag has no such restriction and every request should be attributable.

Folding them into one file would either give the tag that restriction for no reason, or take it away from the token. So both arrays are assembled tag-first with the token appended where allowed, and the off-site hop that deliberately carries no credential still carries the tag.

Verification

Not inferred from the code:

$ curl -v -K <rc> …
> X-Blog-Check: proxmox/wire-proof          <- on the wire

==> tagging requests X-Blog-Check: proxmox/manual              (no CI env)
==> tagging requests X-Blog-Check: github/31291656581-2        (CI shape, attempt 2)
==> tagging requests X-Blog-Check: proxmox/header-verify       (override)

PASS - 1253 URLs honored                                       (full run, staging mirror)

shellcheck and shfmt -d clean. Two tagged requests (proxmox/q-handshake, one 200 and one 404) were sent to production at 2026-08-09T03:47:51Z for the host side to confirm capture from our end.

CHECK_TAG is added to the env-docs gate's KNOBS and described in ENVIRONMENT.md — and the gate was watched failing on the undocumented knob before the row was written.

What this deliberately does not do

Per their §31.4, and worth keeping in the record: it is forgeable, it gates nothing, and it must never reach auth, rate limiting, robots handling, or caching. Its absence is not proof of a human — a scanner sends no header either — so it pairs with the scanner-shape filters rather than replacing them. It is also not retrospective: the traffic already logged has no such field.

[Copilot]

Pull request overview

Adds a synthetic-traffic provenance header (X-Blog-Check) to all requests made by the live URL contract checker, so production logs can reliably distinguish CI/probes from real visitors and attribute failures to a specific run/attempt.

Changes:

• Emit an X-Blog-Check: <source>/<id> header on every checks/check-live-urls.sh request, deriving a default tag from GitHub Actions run ID + attempt or a local proxmox/manual fallback.
• Keep credential and non-credential curl config separated (tag always; token only same-origin), while ensuring the tag is still carried across off-site hops.
• Document the new CHECK_TAG knob and teach the env-docs gate to accept it.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
ENVIRONMENT.md	Documents the CHECK_TAG per-invocation knob and its derived default values.
checks/check-live-urls.sh	Adds unconditional X-Blog-Check header via a curl config file and threads it through same-origin credential handling.
checks/check-env-docs.py	Updates the env-docs validation allowlist to include CHECK_TAG.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

checks/check-live-urls.sh:81

• The CHECK_TAG validation comment/doc says the grammar is <source>/<id>, but the current check only enforces an allowlist of characters and will accept values with no slash (or leading/trailing slash), which undermines the provenance format and makes log queries less reliable. Consider enforcing the exact <source>/<id> shape (one slash, non-empty halves) in the validation itself.

# The grammar is the one the design already states, `<source>/<id>`, kept deliberately narrow:
# letters, digits, dot, underscore, hyphen, and the slash that separates the two halves.
case "$CHECK_TAG" in
*[!A-Za-z0-9._/-]*)
	echo "FAIL CHECK_TAG may contain only letters, digits, and . _ - / -- got '$CHECK_TAG'" >&2

[ptr727]

Fixed in fc81267. Right, and this is the same failure that has come up repeatedly today: a comment describing a rule stricter than the one actually running.

The CHECK_TAG validation comment/doc says the grammar is <source>/<id>, but the current check only enforces an allowlist of characters and will accept values with no slash (or leading/trailing slash).

The charset check passed all of these while every one of them reads as conforming:

smoke       /smoke       proxmox/       a/b/c

Why the shape and not just the characters. The whole value of provenance over a boolean is grouping the log by source — select(.tag | startswith("github/")). That is only reliable if every tag actually has a source half. A tag with no slash, or three, breaks the query quietly rather than loudly, which is the worst shape for something whose entire job is making a log answerable.

Now enforced: exactly one /, both halves non-empty, same narrow character set. Each rule demonstrated rejecting, and both legal shapes demonstrated passing:

smoke                 -> FAIL must be <source>/<id>
/smoke                -> FAIL needs a non-empty half either side of the /
proxmox/              -> FAIL needs a non-empty half either side of the /
a/b/c                 -> FAIL takes exactly one /
proxmox/media dev     -> FAIL character set
proxmox/media-dev     -> accepted
github/999-3          -> accepted

I also re-checked the two derived defaults against the new rule rather than assuming they satisfy it — proxmox/manual and github/<run-id>-<attempt> both pass. Adding a validator after the values it governs is exactly how you ship one that rejects your own defaults.

ENVIRONMENT.md now says the shape is enforced rather than expected. Full run still PASS - 1253 URLs honored; shellcheck, shfmt, markdownlint and the env-docs gate all clean.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

checks/check-live-urls.sh:95

• This failure message also has an extra space before the comma (<source>/<id> ,), which looks like a typo. Tighten punctuation so the diagnostic reads naturally.

*)
	echo "FAIL CHECK_TAG must be <source>/<id> , such as proxmox/media-dev -- got '$CHECK_TAG'" >&2
	exit 2

checks/check-live-urls.sh:86

• The error message has an extra space before the comma (/ ,), which reads like a typo in a user-facing failure path. Remove the stray space to keep diagnostics clean.

This issue also appears on line 93 of the same file.

*/*/*)
	echo "FAIL CHECK_TAG takes exactly one / , as <source>/<id> -- got '$CHECK_TAG'" >&2
	exit 2

[ptr727]

Fixed in 84db125, both instances plus a third of the same shape you did not flag.

The error message has an extra space before the comma (/ ,), which reads like a typo in a user-facing failure path. This issue also appears on line 93.

Both were mine trying to keep a bare / from running into the punctuation after it. Naming the character instead of printing it removes the problem rather than working around it — "takes exactly one slash" and "either side of the slash" have nothing to collide with.

The third instance was the character-set message, where the slash ran into the separator the same way: . _ - / -- got. The set is quoted now.

Read back as a user sees them:

FAIL CHECK_TAG may contain only letters, digits, and the characters '. _ - /' -- got 'proxmox/media dev'
FAIL CHECK_TAG takes exactly one slash, as <source>/<id> -- got 'a/b/c'
FAIL CHECK_TAG needs a non-empty half either side of the slash -- got '/smoke'
FAIL CHECK_TAG must be <source>/<id>, such as proxmox/media-dev -- got 'smoke'

Full run still PASS - 1253 URLs honored; shellcheck and shfmt clean.

Thanks for naming the second occurrence explicitly rather than only the first — that is what made me sweep for the third instead of fixing the two and moving on.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

ENVIRONMENT.md:92

• The CHECK_TAG row says the value is limited to letters/digits and ., _, -, but the enforced format is <source>/<id> (which necessarily includes a single /). Update the wording so the allowed character set matches the script’s validation (slash allowed only as the single separator).

| `CHECK_TAG` | the `X-Blog-Check` provenance this run announces on every request. **`<source>/<id>` is enforced, not merely expected**: exactly one `/`, both halves non-empty, and only letters, digits, `.`, `_`, `-`. Rarely set by hand, since `check-live-urls.sh` derives `github/<run-id>-<attempt>` under Actions and `proxmox/manual` elsewhere. Set it to name a purpose for a hand run, as `proxmox/media-dev` |

[ptr727]

Fixed in cd95e28.

The CHECK_TAG row says the value is limited to letters/digits and ., _, -, but the enforced format is <source>/<id> (which necessarily includes a single /).

Right — the row required "exactly one /" and then described a character set with no slash in it. Read strictly the two clauses contradict each other; read charitably the reader has to guess that the set governs the halves rather than the whole value.

Now:

exactly one /, which is the separator and the only one allowed, with both halves non-empty and each drawn from letters, digits, ., _, -

That is what the script actually does, and it is worth spelling out because the implementation gets there in two steps rather than one: the charset check permits a slash anywhere, and the two rules after it leave exactly one in the separator position — so each half can only contain the rest. The doc now describes the net effect rather than either half of the mechanism.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

checks/check-live-urls.sh:81

• The CHECK_TAG allowlist uses glob ranges (A-Za-z0-9) which are locale-dependent in shell pattern matching. In non-C locales, these ranges can include unexpected characters, weakening the "strict allowlist" guarantee for values written into the curl config file. Prefer an explicit ASCII character class (or force LC_ALL=C) to keep the validation stable across environments.

case "$CHECK_TAG" in
*[!A-Za-z0-9._/-]*)
	echo "FAIL CHECK_TAG may contain only letters, digits, and the characters '. _ - /' -- got '$CHECK_TAG'" >&2

[ptr727]

Fixed in 27cce35, and this one reproduces. I nearly declined it, then measured.

The CHECK_TAG allowlist uses glob ranges (A-Za-z0-9) which are locale-dependent in shell pattern matching.

It does not misbehave here — bash 5.2.37 has globasciiranges on by default, and é, É, µ are all rejected identically under C, en_US.UTF-8 and C.UTF-8. So the observable behaviour was already correct.

But the guarantee was resting on a default the script never set. With the option off, in a UTF-8 locale:

shopt -u globasciiranges     aé -> ACCEPT     aÉ -> ACCEPT
shopt -s globasciiranges     aé -> REJECT     aÉ -> REJECT

So on a shell where it is off, an accented letter would have been written into the curl config while the comment above called the set narrow. shopt -s globasciiranges is now set explicitly, and re-verified against the hostile case — option forced off beforehand, UTF-8 locale, proxmox/café is rejected.

Worth being precise about the consequence, because it is smaller than "weakens the allowlist" suggests and the reason matters. Neither a quote nor a newline can arrive through a widened letter range, and those are the only two characters that break out of a quoted config line — so this was never a path to config injection. What was wrong is the claim: a validator describing itself as strict has to be strict on its own terms rather than on the terms of whatever shell happens to run it.

That is the same failure as the last two rounds — a rule asserting more than it enforced — arriving a third time in a different disguise. Good catch.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

checks/check-live-urls.sh:121

• The Pangolin token guard rejects quotes and LF, but it still allows carriage returns. \r is also invalid in HTTP header values and can participate in header injection/syntax breakage in the generated curl config. Reject CR as well (alongside LF).

	for name in PANGOLIN_ACCESS_TOKEN_ID PANGOLIN_ACCESS_TOKEN; do
		case "${!name}" in
		*'"'* | *$'\n'*)
			echo "FAIL $name contains a quote or a newline, which cannot appear in an HTTP header value" >&2
			exit 2

checks/check-live-urls.sh:85

• shopt -s globasciiranges is relied on to make the [A-Za-z0-9] allowlist ASCII-strict, but the script does not fail if that shopt is unsupported (there is no set -e). On a system with an older bash, the validation can silently fall back to locale-dependent ranges, contradicting the guarantee in the surrounding comment.

This issue also appears on line 117 of the same file.

# The range `A-Za-z0-9` is collation-dependent, so the allowlist below is only ASCII-strict
# because `globasciiranges` happens to be on. Set explicitly rather than inherited, since a
# guarantee resting on a build default is not a guarantee. Demonstrated rather than assumed:
# with the option off, under en_US.UTF-8, `aé` and `aÉ` are both ACCEPTED by this pattern,
# and with it on they are rejected.
shopt -s globasciiranges
case "$CHECK_TAG" in

[ptr727]

Both fixed in af22557. The second one is the more embarrassing and the better catch.

1. CR in the token guard

The Pangolin token guard rejects quotes and LF, but it still allows carriage returns.

Correct, and measured before fixing — a token containing \r was accepted. Header injection is classically CRLF and a lone CR is enough on its own, so refusing LF while allowing CR leaves precisely the shape the guard exists for.

Both variables now reject all three, and I re-ran the LF and quote cases to confirm nothing regressed:

PANGOLIN_ACCESS_TOKEN_ID=$'a\rb'   -> FAIL   (was ACCEPT)
PANGOLIN_ACCESS_TOKEN=$'c\rd'      -> FAIL   (was ACCEPT)
PANGOLIN_ACCESS_TOKEN_ID=$'a\nb'   -> FAIL
PANGOLIN_ACCESS_TOKEN_ID='a"b'     -> FAIL

CHECK_TAG needed nothing here — its allowlist already excludes CR, since CR is not in A-Za-z0-9._/-.

2. The unchecked shopt

shopt -s globasciiranges is relied on to make the allowlist ASCII-strict, but the script does not fail if that shopt is unsupported (there is no set -e).

This is the same defect the previous commit set out to fix, reintroduced one line below it. The script runs under set -uo pipefail and not -e, so on a shell lacking the option the command prints to stderr, returns 1, and execution steps straight over it — leaving the allowlist locale-dependent underneath a comment promising it is not.

I had just written "a guarantee resting on a build default is not a guarantee" and then rested the guarantee on an unchecked command.

Verified the failure is detectable rather than assuming it:

$ shopt -s definitely_not_an_option
bash: shopt: definitely_not_an_option: invalid shell option name
exit=1

So it is now fatal and says why.

Full run still PASS - 1253 URLs honored; shellcheck and shfmt clean.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.