Test the backup and 404 flows, put the pull in git, and describe every config value once

[ptr727]

The maintainer asked for the two mechanisms under Recurring operations to be tested rather than read. Both were. What they found turned into the rest of this branch: the pull script had nowhere to live, its host values had to be retyped by hand, and every configuration value was described in whichever file its author happened to be editing.

The two mechanisms, exercised

The backup, without spending the evidence. systemctl start would have written the second journal entry that 2026-08-09 exists to prove, so the transport was exercised with --dry-run and the schedule was read instead. The journal still holds exactly one entry and the timer's LAST is still -, so tomorrow is genuinely the first time it fires.

The 404 review, end to end. 9,996 edge requests, 9,285 of them this repo's own deploy gate. Of the 711 that remain, 101 were site-host 404s across 73 distinct paths, and every one was a scanner shape. Only / and /robots.txt touch the URL contract and both are explained. Nothing is owed to golden-urls.txt, recorded as a clean pass because an unrecorded one is indistinguishable from a pass nobody ran.

Four things the run found, each written where it binds:

• A referer does not implicate this site unless it points somewhere else. All 36 referer-bearing site-host 404s were scanners setting Referer to the request URL, so the rule as written reported three dozen broken links on a site that had none.
• ServiceName separates an edge 404 from a site 404 inside one log, which the tier table described conceptually and gave no way to apply. 99 of 101 reached the site. The two that did not are the VPS's tier and have been raised with them.
• jq 'select(...)' | wc -l counts lines, not records. It reported 37 and 1,332 where the truth was 1 and 36.
• --dry-run named no files, so the one flag that answers "what will tomorrow bring" could not answer it.

ops/, because the pull had no copy anywhere

The script that makes the off-host copy lived only on the backup host, in /usr/local/bin and an untracked directory beside it — the same machine the copies live on. It is now committed with both units, an installer, and a README.

Nothing in ops/ names a machine. VPS_SSH_HOST, BACKUP_ARCHIVE_ROOT and LOG_ARCHIVE_ROOT have no defaults and preflight refuses to run without them, since a wrong-but-valid destination is a backup nobody can find. User= and RequiresMountsFor= live in a drop-in, and the script refuses to run as root rather than warning about it.

ops/install.sh derives every host value from secrets/<server>.<environment>.env rather than asking for it. Two derivations where the obvious answer is wrong, both caught by running it:

• The group comes from the destination, not id -gn. Here those differ, and Group= sets the process's primary group, so the obvious answer creates files the existing tree's group cannot read — and only where no setgid bit masks it, so it half works.
• RequiresMountsFor= needs the mount point, not the destination below it.

Installed and verified: byte-identical to the committed copy, systemd resolving User, Group and RequiresMountsFor from the drop-in, environment file 600 root:root, journal still one entry.

ENVIRONMENT.md, and a gate so it stays true

Every configuration value is described once, grouped by where it lives, because that determines who can change it and what happens when it is wrong. The .example files stated the format and the scripts state the defaults; both now point here.

The two templates collapse into one example.env. .gitignore needed a negation for it — *.env matched the template, which is what the old .example suffixes had been dodging. Anchored, so a real environment file anywhere is still ignored.

checks/check-env-docs.py runs in CI and fails both directions: a value declared with no row, and a row describing something declared nowhere, which is the shape a rename leaves. Tested by breaking it each way rather than trusting a clean first run.

Corrections to the record

• The pull's two sides had grown four duplicate names for the same directories, needing a mapping table here and a translation step in the installer. Unified, and both are deleted rather than maintained.
• The script's own install instructions pointed at /usr/local/sbin/, which nothing runs. scp and chmod would both report success while the timer kept running the previous copy.
• The archive directory carries MANIFEST.txt as well as RECONCILE.md, so the file count exceeds the number of logs by two rather than the one this PR's first commit recorded.
• SSH_OPTS was unquoted and contains spaces, harmless while only systemd read it and broken once a shell sources the same file. The comment claiming systemd cannot strip quotes was wrong, measured with systemd-run rather than assumed.
• ops/ was not in the shell-lint gate the repo's other two scripts pass. Both are now, formatted and with two shellcheck findings suppressed with reasoning at the site.

Verification

The mirror gate does not apply: no change to deploy/Caddyfile, deploy/maps/, content/, static/, hugo.yaml or layouts/, which OPERATIONS.md exempts explicitly. Every other CI step was run locally through the Docker images GOVERNANCE.md names.

Gate	Result
markdownlint	18 files, 0 issues
cspell	0
actionlint	0
editorconfig-checker	0
shellcheck + shfmt	4 scripts clean
config validation	OK
check-env-docs.py	28 values, all described
hugo --panicOnWarning	clean
check-url-parity.py	PASS, 328 render, 778 media, 1049 assets

The hub's prose_lint.py is at baseline on every file: nothing I authored adds a finding.

[Copilot]

Pull request overview

Updates the repository’s operational documentation to reflect the latest production deploy state and to move/record lessons learned from the off-host log audit in a durable location for future log reviews.

Changes:

• Update TODO.md State/operations notes to reflect the redeploy, /robots.txt behavior, and off-host audit findings/lessons.
• Update OPERATIONS.md “Log Review” guidance to record reconciliation handling (RECONCILE.md) and clarify why service journals alone can’t prove copy history.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
TODO.md	Refreshes migration status records (production release, robots verification) and captures off-host audit lessons in the running backlog.
OPERATIONS.md	Extends log-review operational guidance with rename reconciliation mechanics and copy-timestamp interpretation.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (3)

ops/install.sh:111

• The generated /etc/vps-backup-pull.env sets SSH_OPTS without quotes, but this value contains spaces. That breaks if the env file is ever sourced by a shell (it will try to execute "ConnectTimeout=15" as a command) and may also be parsed unexpectedly by systemd. Quote the value to match example.env and to keep the generated env file safe to reuse.

BACKUP_ARCHIVE_ROOT=$BACKUP_ARCHIVE_ROOT
LOG_ARCHIVE_ROOT=$LOG_ARCHIVE_ROOT
SSH_OPTS=-o ConnectTimeout=15 -o BatchMode=yes
EOF

ops/vps-backup-pull:25

• This comment references CONFIG.md, but there is no CONFIG.md in the repository. That makes the header misleading and sends readers to a dead document; reference the actual doc that explains the pull-vs-push security rationale (e.g., OPERATIONS.md "Working With the VPS").

# Why a PULL and not a push from the VPS: it keeps the VPS free of any credential
# reaching another system, which is the one security rule in CONFIG.md item 0.
# This host already has an SSH key trusted by the VPS; the VPS has none for here.

ops/vps-backup-pull:296

• In the --dry-run early-exit path, the completion summary is printed via say(), so it is suppressed under --quiet. That contradicts the script's "--quiet = errors and summary only" behavior and the tell() change that was added specifically to keep summaries visible under --quiet.

say
if [[ $DRYRUN -eq 1 ]]; then
    say "=== dry run complete -- nothing changed"
    exit 0

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (1)

ops/vps-backup-pull:303

• Because set -euo pipefail is enabled, the du | cut pipelines will terminate the script if du fails, but the 2>/dev/null redirection hides the underlying error. That can turn an otherwise-successful pull into a non-obvious failure right at the summary step. Consider letting du emit its error and failing with an explicit message so the journal shows why the run exited non-zero.

ARCHIVES=$(find "$BACKUP_ARCHIVE_ROOT/pangolin" -name '*.enc' -type f | wc -l)
SIZE=$(du -sh "$BACKUP_ARCHIVE_ROOT" 2>/dev/null | cut -f1)
if [[ $LOGS -eq 1 ]]; then
    LOGSIZE=$(du -sh "$LOG_ARCHIVE_ROOT" 2>/dev/null | cut -f1)
    tell "=== done -- $ARCHIVES archive(s), $VERIFIED verified, $SIZE in $BACKUP_ARCHIVE_ROOT; $LOGFILES log file(s), $LOGSIZE in $LOG_ARCHIVE_ROOT"

[ptr727]

Answering the four suppressed findings, which have no threads to resolve. All four were real and all four are fixed in 5e3ef21. Each was reproduced before being accepted.

---

ops/vps-backup-pull:303 — Because set -euo pipefail is enabled, the du | cut pipelines will terminate the script if du fails, but the stderr redirection hides the underlying error.

Correct, and it is the worst-shaped of the four. Reproduced against a missing path: the script printed before, then exited 1, and the line after the assignment never ran.

So a pull that copied every archive and every log would exit 1 at the summary, having done all its work, and leave a journal with no indication of why. Under the timer that reads as a failed backup.

The size is cosmetic, so the fix keeps the transfer's success rather than the size's: du now reports to stderr where the journal can see it, and the run continues with size unavailable.

---

ops/install.sh:111 — The generated /etc/vps-backup-pull.env sets SSH_OPTS without quotes, but this value contains spaces.

Correct, and worse than it looks: the same defect was fixed in example.env one commit earlier and missed in the file that generates the real one.

Quoting is safe for both readers, measured rather than assumed. A systemd-run unit with an EnvironmentFile containing a quoted value received it with the quotes stripped, identical to the bare form. So systemd is indifferent, and the quoted form is the only one that also survives a shell sourcing the same content.

Note for anyone re-running the installer: /etc/vps-backup-pull.env on the backup host still holds the unquoted form, so ops/install.sh will now report it as differing and stop. That is the intended behavior. Re-run with --force to replace it.

---

ops/vps-backup-pull:296 — In the --dry-run early-exit path, the completion summary is printed via say(), so it is suppressed under --quiet.

Correct. --quiet --dry-run printed no summary at all, which contradicts the flag's own documented contract of errors and the summary. It is the same defect tell() was introduced to fix on the normal summary path, and the dry-run exit was missed. It uses tell() now.

---

ops/vps-backup-pull:25 — This comment references CONFIG.md, but there is no CONFIG.md in the repository.

Correct that it is not in this repository, and the reference was misleading. The document is real but lives on the other host, as /root/CONFIG.md, confirmed present there. It is one of the paths the VPS advertises for the host-config leg, so the pull copies it.

The comment now names the host that holds it and points at OPERATIONS.md "Working With the VPS" for the same reasoning stated locally.

---

Coverage

The review reports 15 of 16 changed files and names no file list, so which file went unread cannot be determined from the API. Flagging rather than re-requesting: on record here a partial has never cleared on a re-request, and it is the maintainer's call.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (5)

ops/vps-backup-pull:22

• This comment implies the script itself reads /etc/vps-backup-pull.env. In reality it only reads environment variables; systemd loads /etc/vps-backup-pull.env via EnvironmentFile, but a direct shell invocation won’t pick it up unless the caller sources/exports it. Tightening this wording avoids confusion when running the script manually.

# Paths below are read from /etc/vps-backup-pull.env, so a host states its own layout
# there rather than editing this file. See example.env and ENVIRONMENT.md.

ops/vps-backup-pull:16

• The script header says to schedule this via a user crontab/user systemd timer ("not root's"), but this PR also provides a system-level systemd timer/service under /etc/systemd/system. That guidance is misleading; the important constraint is that the unit runs as a non-root user via User= (drop-in), regardless of whether it’s a system timer or user timer.

This issue also appears on line 21 of the same file.

# the unit sets User=. Same applies when scheduling it: use the user's crontab or a
# user systemd timer, not root's.

ops/install.sh:55

• ENV_FILE is taken verbatim, so a relative ENV_FILE will resolve against the caller’s current working directory (unlike deploy/make-release.sh, which normalizes relative ENV_FILE against the repo and refuses traversal). That makes it easy to accidentally read the wrong environment file during install/check.

REPO=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
ENV_FILE=${ENV_FILE:-$REPO/secrets/local.production.env}

[[ $EUID -ne 0 ]] || die "do not run this under sudo -- run it as the account that will own the backup; it calls sudo for the steps that need it"
[[ -f $ENV_FILE ]] || die "$ENV_FILE does not exist (ENV_FILE overrides which file is read)"

checks/check-env-docs.py:6

• The docstring says this check fails when a value is “declared or consumed anywhere”, but the implementation only treats templates (example.env), workflow refs, and the explicit KNOBS list as the declared surface. Narrowing the wording avoids over-promising coverage.

"""Fail if a configuration value is declared or consumed anywhere but not described in
ENVIRONMENT.md, or described there and declared nowhere.

ENVIRONMENT.md is the single description of every configuration value. A new value gets
added wherever its author is working, which is rarely the doc, and no linter notices a

ops/README.md:51

• install.sh currently always rewrites /etc/vps-backup-pull.env and the drop-in (via sudo install), even when they already match. This sentence claims matching files are “left alone”, which isn’t accurate (even if the content is identical).

Re-running is safe and is how a changed value is applied. A file that already matches is reported and left alone, and one that differs stops the run until `--force`.

[ptr727]

Second round of suppressed findings, again with no threads to resolve. All five were real and all five are fixed in e7e6b15. That is nine for nine across both rounds, which is worth saying plainly given they arrive marked low confidence.

---

ops/install.sh:55 — ENV_FILE is taken verbatim, so a relative ENV_FILE will resolve against the caller's current working directory (unlike deploy/make-release.sh, which normalizes relative ENV_FILE against the repo and refuses traversal).

Correct, and the comparison to make-release.sh is exactly right — that script normalizes against the repo and rejects a traversing name, and the installer did neither. It is also the worse place to have the gap, because this is the script that writes to /etc and enables a timer.

It now applies the identical rule. Verified both ways: run from /tmp with a relative ENV_FILE, it resolves against the repo and derives the same values; given a traversing name it refuses with ENV_FILE must not traverse.

---

ops/vps-backup-pull:22 — This comment implies the script itself reads /etc/vps-backup-pull.env. In reality it only reads environment variables.

Correct. The script opens no file. systemd loads it through the unit's EnvironmentFile, so a direct run picks up nothing unless the caller sources an environment file first — which is exactly what I had to do to exercise it by hand, without noticing the comment said otherwise. Reworded to say the script reads the environment, name what loads the file under the timer, and state that a manual run has to source one.

---

ops/vps-backup-pull:16 — The script header says to schedule this via a user crontab/user systemd timer ("not root's"), but this PR also provides a system-level systemd timer/service under /etc/systemd/system.

Correct, and the guidance predates what this PR ships. It came across from the copy on the other host, where no unit existed. The constraint that matters is the account, which the drop-in sets with User=, and a system timer satisfies it as well as a user timer does. Reworded to say that rather than to rule out the thing being shipped.

---

checks/check-env-docs.py:6 — The docstring says this check fails when a value is "declared or consumed anywhere", but the implementation only treats templates, workflow refs, and the explicit KNOBS list as the declared surface.

Correct, and over-promising in a docstring is worse than usual here, because the whole point of the check is that nobody should have to remember what it covers.

Widening the implementation is not the fix, though, and the docstring now says why: a script-local variable and a configuration value are the same shape, so scanning for reads would report the difference as findings nobody can clear. The boundary is now stated explicitly, including that a new knob has to be added to KNOBS by hand.

---

ops/README.md:51 — install.sh currently always rewrites /etc/vps-backup-pull.env and the drop-in (via sudo install), even when they already match. This sentence claims matching files are "left alone", which isn't accurate.

Correct. Both generated files are written unconditionally in the install path. The comparison is a report and the --force guard, not a skip, and the README now says that instead of describing behavior the code does not have.

---

Coverage

Still 15 of 16 files with no file list, so the unread file cannot be identified from the API. Reporting rather than re-requesting, per the same reasoning as the previous round.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.