diff --git a/OPERATIONS.md b/OPERATIONS.md index 22b83b8..fe3d680 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -265,7 +265,15 @@ Two properties of the Caddy side are worth knowing before parsing it. Its access The outward pass is four filters over the edge log, and each one exists because skipping it produced a wrong answer once. -**Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. Filter on user agent: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0` and the 711 that remained are the entire real dataset. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large. +**Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large. + +The mechanism is the `X-Blog-Check` request header, which the scripted checks send on every request they make, so `jq 'select(.["request_X-Blog-Check"] == null)'` is the filter. It is only as complete as the tagging is, which is the first bullet below. Its value is a source and an id rather than a boolean, so a run is identifiable rather than merely excludable, and real values look like `github/31322640628-1` from CI, `vps/smoke` from the host side, and `proxmox/media-dev` from here. The shape is enforced by `check-live-urls.sh`, which takes exactly one `/` and only letters, digits, `.`, `_`, `-`, so a placeholder written with angle brackets is a description rather than something to paste. + +- **The field exists only because the edge is configured to log that header**, which is the host side's to hold and not this repository's. An absent field therefore has two meanings, an untagged request or a capture that stopped, and they are not distinguishable from the log alone. Confirm the capture is live before reading a day's absence as a day of real traffic. + +- **A hand probe carries it only because whoever runs it adds it.** `check-live-urls.sh` sends it on every request and a bare `curl` sends nothing, so an interactive probe passes `-H "X-Blog-Check: proxmox/media-dev"`. The source half stays `proxmox`, which is where the probe came from, and the id half is where the purpose goes. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window. +- **Absence is not proof of a human**, since a scanner sends no header either, so this pairs with the scanner-shape filters below rather than replacing them. The field is forgeable and must never reach auth, rate limiting, robots handling, or caching. +- **Before 2026-08-09 the log carries no such field**, and user agent is the only key for those days: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0`, leaving 711 real ones. That key is a coincidence rather than a rule, since the CI runner's curl and the host's are byte-identical and only the rotating client address separates them, which is why the header exists. **A referer does not implicate this site unless it points somewhere else.** The rule worth applying is that a 404 carrying a referer is a broken link and a 404 without one is a typed or probed address, and it fails on scanners, which set `Referer` to the request URL itself. Every one of the 36 referer-bearing site-host 404s on 2026-08-08 was self-referential, so the unrefined rule reported three dozen broken links on a site that had none. Discard the matches before counting, and **normalize the scheme rather than comparing it**, because a scanner reaching an HTTPS site routinely sends an `http://` referer for the same address. Comparing against the request's own scheme therefore matches nothing and leaves every false positive in place: on the 2026-08-08 data the naive form kept all 36 where the normalized form kept none. @@ -291,7 +299,7 @@ Widen `== 404` to `>= 400` for the whole non-200 sweep the table above describes ### Retention Is the Prerequisite, and It Belongs to the Host -**On the VPS the reviewable record is Traefik's access log**, at `/var/log/traefik/access.log`, one JSON object per line, one line per request, across every hostname the host serves. `RequestPath` carries the query string, so the legacy `/?p=` traffic is visible as itself. Request headers are dropped except `Referer` and `User-Agent`, which is what keeps the Pangolin resource access token out of a file that is retained and copied, and query strings are logged in full, so treat an extract as sensitive. +**On the VPS the reviewable record is Traefik's access log**, at `/var/log/traefik/access.log`, one JSON object per line, one line per request, across every hostname the host serves. `RequestPath` carries the query string, so the legacy `/?p=` traffic is visible as itself. The log records only three request headers, which appear as the keys `request_Referer`, `request_User-Agent`, and `request_X-Blog-Check`, and omits the rest from the record rather than stripping them from the request, which still arrives intact. That omission is what keeps the Pangolin resource access token out of a file that is retained and copied, and query strings are logged in full, so treat an extract as sensitive. That keep-list is the host's to hold and is the precondition the outward pass depends on: a header absent from it does not appear in the log at all, which is indistinguishable from a request that never sent one. **That log rotates and is eventually deleted, on a schedule the host sets and can change.** The window is long, and it is finite, so anything the inward pass depends on has to be copied off the host before the archive ages out. Read the current retention from the host rather than from this file, because a number written here is a number nothing checks. diff --git a/TODO.md b/TODO.md index 055fe2f..86634c0 100644 --- a/TODO.md +++ b/TODO.md @@ -19,8 +19,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | Fleet conformance | cataloged in the hub registry, audited, and carrying the current canonical | | Deploy pipeline | `deploy-site.yml` is dispatchable and has deployed staging from CI end to end, through a transport retested against the real host | | VPS staging | live at `blog.vps.insanegenius.net`, behind the auth gate, serving a pipeline release | -| VPS production | serving release `20260809-030521` at `blog.insanegenius.net` as of 2026-08-09, deployed from `main` after the promotion and read back from the live `X-Blog-Release` header rather than from the run's exit code. `200` unauthenticated, `X-Robots-Tag: noindex, nofollow` still set for the rehearsal, and the live check's 1,253 URLs verified against the running site, which is the 328 + 917 + 8 subset described in the URL contract row rather than a different contract. That release is **the first built with the mtimes restored**, so the first whose static tree can arrive as hard links, and therefore the first where a bad file mode could ride one forward into every later release. The shared-inode count is server-side, so ask the host for it rather than inferring it here. **M7a, the first production deploy, was 2026-08-08 on release `20260808-154717`**, which established the unauthenticated `200`, the `.net` sitemap line in `/robots.txt`, and the gallery fix. DNS for the public name is still on the old platform | -| Operations | started, and neither half has completed a **scheduled** run. The off-host log pull is installed, armed for 09:00 UTC daily, and has copied once, started by hand, so the timer itself has never fired and 2026-08-09 is its first scheduled run. The periodic log review has run once by hand, the outward pass only: it read 2026-08-08 traffic and found nothing to add to the URL contract, the inward pass has not run, and neither pass is on the cadence **Recurring operations** sets | +| VPS production | serving release `20260809-163217` at `blog.insanegenius.net` as of 2026-08-09, deployed from `main` and read back from the live `X-Blog-Release` header rather than from the run's exit code. `200` unauthenticated, `X-Robots-Tag: noindex, nofollow` still set for the rehearsal, and the live check's 1,253 URLs verified against the running site, which is the 328 + 917 + 8 subset described in the URL contract row rather than a different contract. **Hard-linking works and the host measured it**: 1,052 of 3,275 files shared with the previous release, the linked set being exactly the restored generation, and all 1,048 linked media files at mode `644`. A release now costs 11.4 MB incremental against 585 MB for a full copy. Staging reproduced the same 1,052 independently. **M7a, the first production deploy, was 2026-08-08 on release `20260808-154717`**, which established the unauthenticated `200`, the `.net` sitemap line in `/robots.txt`, and the gallery fix. DNS for the public name is still on the old platform | +| Operations | the backup half is proven on its schedule; the review half is not. **The off-host log pull fired unattended on 2026-08-09 at 09:11:01 UTC**, inside its randomized window, proven by both halves of the standard: a second journal entry and `LAST` moving off `-`. It copied more than the hand run, 52 archives against 42. The periodic log review has run once by hand, the outward pass only: it read 2026-08-08 traffic and found nothing to add to the URL contract, the inward pass has not run, and neither pass is on the cadence **Recurring operations** sets | ## Blocked on the maintainer @@ -30,8 +30,8 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y ## Next, in dependency order - **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way. -- **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` answers `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07, with the serving release read from the `X-Blog-Release` header rather than from a pipeline's exit code. **It serves `20260809-030521` today; M7a itself was `20260808-154717`**, and the State table above is the current value, so read it there rather than from this paragraph. The host side verified that first production release independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": - - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** +- **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` answers `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07, with the serving release read from the `X-Blog-Release` header rather than from a pipeline's exit code. **M7a itself was `20260808-154717`. What it serves today is in the State table above and is deliberately not repeated here**, since a release id moves with every deploy and a second copy of it goes stale by the afternoon. The host side verified that first production release independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": + - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run the whole live check at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. - **`robots.txt` is decided, built, and deployed, 2026-08-08.** The site emits one, `enableRobotsTXT` is set, and the theme's template derives the `Sitemap:` line from the built `baseURL`, so it names `.net` during the rehearsal and `.com` after the cutover with nothing to remember at M7b. `/robots.txt/` redirects to the real file rather than to the home page, and `check-url-parity.py` gates all of it. Verified from the served bytes on release `20260808-154717`: `/robots.txt` answers 200 advertising `https://blog.insanegenius.net/sitemap.xml`, `/robots.txt/` 301s to it, and `sitemap.xml` carries 312 `.net` URLs and zero `.com`. The record below is kept because the reasoning is what the next decision about crawl directives will need. @@ -93,7 +93,7 @@ The reference leaf the hub now ships carries one step this repo's deploy does no ## Open decisions -- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. **Installed 2026-08-08 with `ops/install.sh`**, which derives the address, both destinations, the account, the group and the mount from `secrets/local.production.env`, so nothing is typed twice. Verified after the fact rather than from the installer's own output: `systemd` resolves `User=pieter`, `Group=users` and `RequiresMountsFor=/data/backup` from the drop-in, and the environment file is `600 root:root`. The running script is byte-identical to the committed one. Re-running the installer after the shell-gate reformat also exercised its idempotent path, which reported both config files already correct and replaced only the script, so a changed value is applied by running it again rather than by editing anything on the host. The root guard was exercised and refused. **The journal still holds exactly one entry and the timer's `LAST` is still `-`**, so installing did not spend the evidence that 2026-08-09 is the first scheduled run. Separately, [#53][issue-53] reconciles the VPS's older copy in both directions rather than by overwriting either. +- **Resolved for the backup pull, 2026-08-08: it is in this repository at [`ops/`](./ops/).** The script, both `systemd` units, an `EnvironmentFile` template naming every path it uses, and a README covering what it does and how to check it. [`OPERATIONS.md`](./OPERATIONS.md) "Working With the VPS" names it and states which of its variables pair with which of this repo's. The reasoning below stands as the record of why, and the same question is still open for everything under it. **Installed 2026-08-08 with `ops/install.sh`**, which derives the address, both destinations, the account, the group and the mount from `secrets/local.production.env`, so nothing is typed twice. Verified after the fact rather than from the installer's own output: `systemd` resolves `User=pieter`, `Group=users` and `RequiresMountsFor=/data/backup` from the drop-in, and the environment file is `600 root:root`. The running script is byte-identical to the committed one. Re-running the installer after the shell-gate reformat also exercised its idempotent path, which reported both config files already correct and replaced only the script, so a changed value is applied by running it again rather than by editing anything on the host. The root guard was exercised and refused. Installing did not spend the evidence for the first scheduled run, which then arrived: **the timer fired unattended on 2026-08-09 at 09:11:01 UTC**, with a second journal entry and `LAST` off `-`. Separately, [#53][issue-53] reconciles the VPS's older copy in both directions rather than by overwriting either. - **Where the rest of the operational tooling lives, given that today it lives nowhere.** `vps-backup-pull`, its `systemd` units, and the environment variables naming both ends of the copy were an operational asset built from another agent's instructions, and they existed only on the Proxmox host. That host is the machine the backup runs *from*, so losing it loses both the copies and the means of making them, and the instructions that produced them are in a channel file this repository deliberately does not carry. Two candidate homes, and the choice is open: **here**, beside the deploy tooling the same host runs, or **the home-automation config repository**, with the rest of that host's configuration. The argument for the second is that nothing about the pull is specific to this site; the argument for the first is that [`OPERATIONS.md`](./OPERATIONS.md) "Log Review" is the thing that stops working without it. - **The pull itself is resolved and the reasoning is kept because it applies to everything still listed here.** What made it urgent was measured: the copy protected everywhere was the VPS's older one, while the copy that actually ran, carrying the log leg the review depends on, was in no snapshot and no repository. Committing it is what closed that, not the backup host's own off-site copy, which never reached the script. - **The directory holding it is named as though it were disposable.** `~/vps-backup-pull-patch` reads as a patch staged against a source, and there is no source: it is the most complete copy of the script in existence. A directory named for a temporary artifact is the one a cleanup deletes, and nothing here would notice until a restore produced the wrong script. @@ -157,7 +157,11 @@ Each of these was hit or nearly hit, and each is cheap to re-trip. - **Do not name any workflow `build-*-task.yml`** while the repo declares `source-only`, since `detect` is literally `["no build-*-task.yml"]`. - **Do not edit `.markdownlint-cli2.jsonc`, `repo-config/configure.sh`, or the two ruleset payloads.** They are carried verbatim and byte-matched against the hub. Scope a glob in the workflow instead. A reviewer finding a real defect in one of them is answered by declining locally and filing it at the hub, never by editing the file to satisfy the review. - **The hub's `main` can promote while a convergence pull request is open**, so ground truth moves underneath work that was correct when it started. It happened twice in one session on 2026-08-03, and the second time added drift the branch could not have known about. Re-run the audit against the hub ref actually carried before claiming convergence, and name that ref in the change, or the claim ages into a false one. -- **A query that matches nothing reads as a clean result.** It has cost three separate false passes: a review-thread poll that could not see suppressed findings, a reviewer filter written in the wrong API's login form, and an audit loop whose `jq` path had moved. Each returned empty, and empty looked like nothing to report. Assert the query matched before reading what it returned, which is what `jq -e` and a non-empty check are for. +- **Before believing a negative, establish that the check could have produced a positive.** An empty result and a clean result are the same bytes, so the question is never "did it find anything" but "could it have". The instances below are the same failure in different costumes, some of which cost a false pass and one of which has not happened yet. It is stated as a demand for evidence rather than as a warning to be careful because *be careful* has no step to perform and *prove the check can fire* does. + - **A query that cannot see its target.** A review-thread poll blind to suppressed findings, a reviewer filter written in the wrong API's login form, an audit loop whose `jq` path had moved, and a journal grep for `Started` from an account not in `adm` or `systemd-journal`, where the lines exist and are not shown to it. Assert the query matched before reading what it returned, which is what `jq -e` and a non-empty check are for. + - **A rule naming a target that does not exist.** `.gitattributes` pinned two paths this repository has never carried, and the comment above them claimed the case was covered, so the one file that needed the pin sat unpinned twenty lines up. [`checks/check-eol-pins.py`](./checks/check-eol-pins.py) gates that file now; the shape is not gated anywhere. + - **A filter whose precondition is unstated.** The outward pass reads an absent `X-Blog-Check` as a visitor, and the field exists only because the edge is configured to log it, so a stopped capture reads as a day of real traffic. Recorded in [`OPERATIONS.md`](./OPERATIONS.md) beside the filter. + - **A tool that reports success having done nothing.** `git-restore-mtime` v2022.12 printed `1,052 files to be processed`, processed none, and exited 0 for four releases. The remedy is the same in kind: assert the outcome rather than the exit status, which is what both release paths now do. - **The hub authors `scripts/pr_review.py`, and hand-rolling the review loop re-discovers its bugs.** One `status` call reports rounds, head coverage, unresolved threads, suppressed findings across every round, and whether a request was ever picked up. `wait` runs the backoff in-process, so a review wait costs one turn rather than one per poll. It is read-only by design and the mutations stay explicit, so fetch and run it rather than reimplementing it. Its README documents the traps below as the reason it exists. - **A review request can sit forever without being picked up, which looks exactly like patience.** Copilot raises a `copilot_work_started` timeline event within about half a minute of accepting; a request that never draws one is not slow, it is inert, and elapsed time cannot tell them apart. The event is REST-only. Recover by clearing the request with `union: false` and an empty `botIds`, then requesting again, after reading the pending set so a human reviewer is not dropped. - **The Copilot reviewer's login differs by API, and a wrong-form filter reads as a clean review.** REST reports `copilot-pull-request-reviewer[bot]`, GraphQL omits the suffix. A filter written in the other form matches nothing, and an empty result is indistinguishable from no findings. Assert the filter matched before trusting what it returned. diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index e37f12d..feb66b3 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -114,7 +114,7 @@ echo "==> tagging requests X-Blog-Check: $CHECK_TAG" # A resource access token opens the proxy's auth gate. # It goes into a curl config file because bash cannot export an array to the parallel checks. -# A command line is also world-readable in ps output, and this runs 1,245 of them. +# A command line is also world-readable in ps output, and every request would carry it. if [ -n "${PANGOLIN_ACCESS_TOKEN_ID:-}" ] && [ -n "${PANGOLIN_ACCESS_TOKEN:-}" ]; then # Same hazard as CHECK_TAG above and the same reason, but a narrower rule, because the # grammar of a credential is the issuer's to define and not this script's. Only the @@ -269,7 +269,7 @@ export BASE FAILED CURLRC CHECKRC echo "==> $BASE" -# One request before the 1,245, because an auth gate turns a bad credential into a total failure. +# One request before the rest, because an auth gate turns a bad credential into a total failure. # Otherwise the output reads as a vanished site rather than a wrong token. # Transport failures are separated from HTTP ones, since a name that does not resolve otherwise # reports as a status code and gets diagnosed as a credential or a symlink.