diff --git a/.github/workflows/deploy-site-task.yml b/.github/workflows/deploy-site-task.yml index 5964693..f0ca4a9 100644 --- a/.github/workflows/deploy-site-task.yml +++ b/.github/workflows/deploy-site-task.yml @@ -77,12 +77,12 @@ jobs: # One update for the job, because each one is a network round trip that can fail on its # own. REQUIRE_BROTLI later makes a missing brotli fatal, so this keeps the build from - # failing, and git-restore-mtime is what the next step runs. + # failing. - name: Install build tools step run: | set -Eeuo pipefail sudo apt-get update - sudo apt-get install --yes --no-install-recommends brotli git-restore-mtime + sudo apt-get install --yes --no-install-recommends brotli # Git stores no mtimes, so a checkout stamps every file with the moment it was written. # The deploy uploads with --link-dest against the previous release, and a file only links @@ -100,12 +100,42 @@ jobs: # began with, so a media file that acquires a bad one stays present, correctly named and # unreadable, through every later release. The live check is what notices that, by # requesting images and failing on the 403. - # `git restore-mtime`, the subcommand form, because the package installs into git's - # exec-path at /usr/lib/git-core rather than onto PATH, so the bare name does not resolve. + # The action rather than the Ubuntu package, because the package is git-tools v2022.12 and + # that release shells out to `git whatchanged`, which current git refuses to run without + # `--i-still-use-this` - a flag there is no way to pass through. It failed twelve times in + # one step, restored nothing, and exited 0, so every release since #65 was a full copy + # while CI reported success. Upstream replaced whatchanged with `git log` in v2025.08, and + # this action vendors exactly that version. - name: Restore file mtimes step + uses: chetan/git-restore-mtime-action@d186aca54f8760da4dec55313195e51ed3ebb0b3 # v2.3 + with: + args: static + + # The step above is the second tool to claim it restored these and not have done it, so + # the outcome is asserted rather than the tool trusted. This is the check whose absence + # let #65 ship broken: `git restore-mtime` printed "1,052 files to be processed" and then + # processed none of them, which reads exactly like success. + # + # The discriminator is that a restored file cannot be newer than the commit it was dated + # from, so no file under static/ may be newer than HEAD's own commit time. A checkout + # necessarily happens after the commit it checks out, so an unrestored tree is always + # newer than that bound and a restored one never is. It calibrates itself from the + # repository, so no measured constant goes stale as content moves. + # + # Counting distinct mtime days looks like the obvious check and is wrong: git restores + # the LAST COMMIT time, and static/ arrived in a bulk import, so a correctly restored + # tree here has exactly one distinct day. That version was written first and rejected + # only because it was run against a real restored clone before being trusted. + - name: Assert mtimes were restored step run: | set -Eeuo pipefail - git restore-mtime static + bound=$(git log -1 --format=%ct) + newest=$(find static -type f -printf '%T@\n' | sort -n | tail -1 | cut -d. -f1) + echo "static/ newest mtime $newest, HEAD committed $bound, margin $((bound - newest))s" + if [ "$newest" -gt "$bound" ]; then + echo "::error::mtime restore did nothing: static/ carries files newer than HEAD's commit, so they still hold their checkout time and --link-dest will link nothing." + exit 1 + fi # The pin lives in the action, so the deploy and validation cannot install different generators. - name: Install Hugo step diff --git a/ENVIRONMENT.md b/ENVIRONMENT.md index c9bfdd9..700d172 100644 --- a/ENVIRONMENT.md +++ b/ENVIRONMENT.md @@ -89,6 +89,7 @@ Set on the command line for one run rather than stored anywhere. | `NO_LINK_DEST=1` | full copy instead of hard-linking from the previous release | | `KEEP_RELEASES` | how many releases `make-release.sh` leaves behind | | `EXPECT_RELEASE` | the release id `check-live-urls.sh` requires the live site to report, which is what makes a rollback verifiable rather than merely exiting zero | +| `CHECK_TAG` | the `X-Blog-Check` provenance this run announces on every request. **`/` is enforced, not merely expected**: exactly one `/`, which is the separator and the only one allowed, with both halves non-empty and each drawn from letters, digits, `.`, `_`, `-`. Rarely set by hand, since `check-live-urls.sh` derives `github/-` under Actions and `proxmox/manual` elsewhere. Set it to name a purpose for a hand run, as `proxmox/media-dev` | ## Two credentials to the VPS, and why they are separate diff --git a/TODO.md b/TODO.md index e5dc8ec..055fe2f 100644 --- a/TODO.md +++ b/TODO.md @@ -10,15 +10,16 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | --- | --- | | Content and media | done. Carries every migrated post and page, with the media hash-verified against the export tar | | Media integrity | closed for the files themselves. The conversion's only loss was 19 images, inside five galleries, restored from the capture, and the 98 files with no page links are adjudicated rather than unknown, with `ORPHANED_MEDIA` failing if that count moves either way. How media was *presented* was a separate loss, found later and now gated | -| URL contract | done. 328 render, 917 redirect, 778 legacy image URLs, all gated | +| URL contract | done. 328 render, 917 redirect, 778 legacy image URLs, all gated. **Two gates read different subsets of that, which is why two totals appear in this file**: the build gate proves all three against files on disk, and the live check requests 328 + 917 + 8 curated media = **1,253** against a running server, since proving 778 images over the wire on every deploy buys nothing the build gate has not already proven | | Deploy shape | done. Proven on two local mirrors and on the VPS, by hand and by pipeline | -| CI workflows | green. Validation runs on every pull request and feeds the required check | +| CI workflows | green. Validation runs on every pull request and feeds the required check. Three gates live in `checks/`: the URL contract, the environment-docs pair, and the line-ending pins | +| Branches | converged 2026-08-09. [#68][pr-68] promoted `develop` to `main` as a merge commit, `main` carries every commit on `develop`, and the content diff is empty. **The merge itself published nothing**, which is the release model behaving as specified: a human merge never auto-publishes, and the releases named two rows down were cut afterwards by explicit dispatch | | GitHub repo | public, both rulesets active, `configure.sh check` exits 0 | -| Release pipeline | proven end to end. `1.0.17-g4b2def3ee9` is the newest, a prerelease from `develop` | +| Release pipeline | proven end to end and cut from both branches 2026-08-09. `1.0.54` is the newest release, from `main` and marked Latest, and `1.0.54-gcd9285a9fe` the newest prerelease, from `develop`. Each tag is pinned to its own branch tip, and the clean-versus-suffixed classification is the gate working rather than a coincidence | | Fleet conformance | cataloged in the hub registry, audited, and carrying the current canonical | | Deploy pipeline | `deploy-site.yml` is dispatchable and has deployed staging from CI end to end, through a transport retested against the real host | | VPS staging | live at `blog.vps.insanegenius.net`, behind the auth gate, serving a pipeline release | -| VPS production | **M7a done 2026-08-08.** Serving release `20260808-154717` at `blog.insanegenius.net`, answering `200` unauthenticated, deployed from `main` by pipeline with the 1,245-URL contract verified against the live site. `/robots.txt` answers 200 carrying a `.net` sitemap line, and the gallery fix is live. DNS for the public name is still on the old platform | +| VPS production | serving release `20260809-030521` at `blog.insanegenius.net` as of 2026-08-09, deployed from `main` after the promotion and read back from the live `X-Blog-Release` header rather than from the run's exit code. `200` unauthenticated, `X-Robots-Tag: noindex, nofollow` still set for the rehearsal, and the live check's 1,253 URLs verified against the running site, which is the 328 + 917 + 8 subset described in the URL contract row rather than a different contract. That release is **the first built with the mtimes restored**, so the first whose static tree can arrive as hard links, and therefore the first where a bad file mode could ride one forward into every later release. The shared-inode count is server-side, so ask the host for it rather than inferring it here. **M7a, the first production deploy, was 2026-08-08 on release `20260808-154717`**, which established the unauthenticated `200`, the `.net` sitemap line in `/robots.txt`, and the gallery fix. DNS for the public name is still on the old platform | | Operations | started, and neither half has completed a **scheduled** run. The off-host log pull is installed, armed for 09:00 UTC daily, and has copied once, started by hand, so the timer itself has never fired and 2026-08-09 is its first scheduled run. The periodic log review has run once by hand, the outward pass only: it read 2026-08-08 traffic and found nothing to add to the URL contract, the inward pass has not run, and neither pass is on the cadence **Recurring operations** sets | ## Blocked on the maintainer @@ -29,7 +30,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y ## Next, in dependency order - **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way. -- **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` serves release `20260808-154717`, answering `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07, read from the `X-Blog-Release` header rather than from a pipeline's exit code. The host side verified the first production release independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": +- **Production is deployed, which the VPS agent calls M7a, done 2026-08-08.** `blog.insanegenius.net` answers `200` unauthenticated on a Let's Encrypt certificate issued 2026-08-07, with the serving release read from the `X-Blog-Release` header rather than from a pipeline's exit code. **It serves `20260809-030521` today; M7a itself was `20260808-154717`**, and the State table above is the current value, so read it there rather than from this paragraph. The host side verified that first production release independently, 9/9 unauthenticated with the built `baseURL` read from the deployed bytes rather than from this repo's config, across a 3,095-request gate run with no unexplained 404s. What remains is **M7b, the `.com` cutover**, and the sub-items below are where this repo stands against it, two of them owed and one already answered. The VPS agent's §19, §20, §23 and §24 carry the detail and that file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": - **`HUGO_BASEURL` on the `production` environment is set to `https://blog.insanegenius.net/`**, done 2026-08-07. It held `https://blog.insanegenius.com/`, the live WordPress address, which is what the workflow both builds with and points the live check at, so a deploy would have baked the old platform's address into every canonical tag, feed link and `sitemap.xml` and then run 1,245 requests at the live site to verify it. **Setting it back to `.com` at M7b is the other half and is not done.** - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. - **The two questions in §19.3 are answered.** `HUGO_BASEURL` holds the interim `.net` name, per the item above. Exactly one place hardcodes `blog.insanegenius.com`: `baseURL` on line 1 of `hugo.yaml`, which is the production default every environment overrides through `HUGO_BASEURL`. Nothing under `checks/`, `deploy/`, `layouts/`, or `.github/` carries it. @@ -41,10 +42,22 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **A wrong `HUGO_BASEURL` is still invisible to every gate here, and the `Sitemap:` line does not change that.** Worth stating because the opposite is easy to believe: the parity check compares the advertised origin against the one on the home page's canonical link, and both come from `baseURL`, so they agree whenever the build is coherent — including when `baseURL` was wrong for the environment. Nothing inside the artifact can see it, which is why the check belongs on the side that knows which host it is serving, and the VPS side does it by reading the origin out of the deployed `sitemap.xml`, `og:url` and `feed.xml`. What the comparison does catch is an origin **written rather than derived**, a committed `static/robots.txt` shadowing the template being the way that happens. - **Media is checked live now, which unblocks the item below.** [`checks/golden-media-live.txt`](./checks/golden-media-live.txt) is fetched by `check-live-urls.sh` against a running server, covering both media trees and the `@uploads` rule, and asserting status, a non-zero body and an image content type so that a 403 from a bad mode, a 404 from a lost transfer, a truncated file and a soft-404 error page are each caught. Verified against production, and each of the four failure shapes was reproduced rather than assumed. The record of why it was needed follows. - **~~Nothing checks that media survived the trip to the server.~~ Closed 2026-08-08, by the item above.** The VPS agent noticed in §24.3 that a 3,095-request gate run fetched no image at all, and asked whether `golden-media-legacy.txt` is wired in. It is, but only at build time, in `check-url-parity.py`, against files on disk. The live check requests pages and redirects and never an image, so a media tree lost **between the build and the server**, a partial upload, is caught by neither: the build passed before the loss and the live gate never asks. On a site whose value is eighteen years of posts with images in them, that is the gap worth closing rather than the one that was suspected. A handful of media URLs in the live check would close it, chosen to cover both trees rather than to be exhaustive, since the build gate already proves the set. The mechanism that makes this concrete rather than theoretical is the hard-link trap below: a link carries its inode's mode, so a media file that acquires a bad one rides the chain into every later release, present and correctly named and unreadable to the server, which `is_file()` on the runner cannot see and a check that never requests an image cannot either. -- **Restore file mtimes in CI so `--link-dest` links. The media check it waited on now exists, so this is unblocked.** The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. +- **~~Restore file mtimes so `--link-dest` links.~~ Done, but by [#75][pr-75] and [#76][pr-76] rather than by [#65][pr-65], which shipped it broken.** CI restores with `chetan/git-restore-mtime-action`, pinned by SHA, and [`deploy/make-release.sh`](./deploy/make-release.sh) requires the tool on `PATH` or as a git subcommand at **v2025.08 or newer**. Both then assert the outcome, since a checkout always happens after the commit it checks out: **no tracked, unmodified file under `static/` may be newer than HEAD's own commit.** CI compares every file under `static/`, because a fresh checkout has nothing uncommitted in it; the local path walks the same directory but excludes modified and untracked paths, which a working tree can legitimately hold newer than any commit, so the check stays usable during an edit loop rather than being skipped whenever the tree is dirty. + - **#65 never restored anything and exited 0 for four releases.** It installed the Debian and Ubuntu package, git-tools v2022.12, which shells out to `git whatchanged`; current git refuses that without `--i-still-use-this`, a flag there is no way to pass through. It printed `1,052 files to be processed`, processed none, and succeeded, twelve refusals inside one green step. **Release `20260809-030521` did not exercise it**, an earlier version of this entry said it did, and the host measured the truth as 0 of 3,275 shared inodes. Local git still permits `whatchanged`, which is why #65's two-clone measurement was honest and measured everything except the machine that runs it. + - **Verified end to end on the local production mirror, 2026-08-09**, with the tool installed by hand: `1,052 of 3,269 files hard-linked`, and the release carrying `site/media` and `site/external` stamped `2026-08-01`, the commit date rather than the build date, which is what proves the restore reached the release and not merely the working tree. Two releases cost 584 MB then 18 MB. + - **The first restored release cannot link, by construction**, because it is compared against a predecessor built with unrestored mtimes. `make-release.sh`'s zero-shared-files guard refuses that one, so the changeover takes a single `NO_LINK_DEST=1` release to seed a restored generation. The deploy has no such guard and will simply link nothing for one deploy. **So the first deploy after the promotion is expected to link 0**, and the one after it is the real test. + - **The shared-inode count on the VPS is still the open half** and is the host's to measure, since nothing here can read an inode there. Told to them in the channel as §S, which corrects §R for predicting the jump one deploy too early. Roughly 1,052 of 3,275 is what the change predicts, at the *second* restored deploy. + - Making `--link-dest` effective is also what makes a file that acquires the wrong permissions able to ride a link into every later release, which is the trap below. The live media check landed first, deliberately, and is what watches for exactly that. + - **The original diagnosis, kept as written on 2026-08-08 because it is what the next change to the deploy's linking behavior will need.** Read it as the record of why the change was made, not as current state: the "today" in it is that date, and the fix it names is #65's, which turned out not to run. The host side measured zero shared inodes across every release the pipeline has delivered, against 1052 of 3266 on a release built here, and the cause is neither the call site nor the confined rsync: both were tested there and link correctly through a relative symlink. Git stores no mtimes, so a CI checkout writes all 3,272 files inside a 23-second window and the `static/` tree that would otherwise match arrives freshly stamped with everything else. `git-restore-mtime` is the fix and needs no checkout change, since `deploy-site-task.yml` already uses `fetch-depth: 0`, and it is deterministic across runs in exactly the place that matters, because `static/` has stable last-commit times. **The ordering is the part worth writing down.** Today every file arrives as a fresh inode, so `--no-g --chmod=D2755,F644` re-establishes the mode contract on every deploy; make the mtimes honest and about a third of the tree starts arriving as links carrying whatever mode its chain began with, which is the trap above. Harmless as things stand, since every inode in the current chain was made by that same rsync line, and it means the live media check should exist first. Nothing is broken meanwhile: the cost is ~585 MB a release, which the host's prune timer reclaims. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. -- **Publish a release from `main`, once the pipeline has soaked.** `1.0.11` is the newest release from `main` and was cut on 2026-08-01, ahead of every deploy change, so the next one is the first that would describe a site actually serving its public address. The mechanism is proven and is not what this waits on: it waits on the switchover being trusted rather than merely green, which is what the log review under **Recurring operations** establishes and no gate can. A release cut before that names a state that has not held yet. +- **~~Publish a release from `main`, once the pipeline has soaked.~~ Done 2026-08-09, with the maintainer lifting the soak condition explicitly.** `1.0.54` is cut from `main` and marked Latest, and `1.0.54-gcd9285a9fe` from `develop`, each tag pinned to its own branch tip. Before this, `1.0.11` had been the newest from `main` since 2026-08-01, predating every deploy change. **The soak reasoning is kept because it was not satisfied, it was overridden**, and the same argument returns at the next release: what a release from `main` cannot yet claim is a switchover that has been *trusted* rather than merely green, which the log review under **Recurring operations** establishes and no gate can. `1.0.54` therefore names a working pipeline and a rehearsal host, not a completed cutover. +- **Gate the deploy ref by environment in both directions, so a deploy cannot do what a release structurally cannot.** [`deploy-site.yml`](./.github/workflows/deploy-site.yml)'s `assert-ref` job refuses production from any ref but `main`, and **staging accepts any ref at all**. Add the missing arm so staging accepts only `develop`: `develop` deploys staging, `main` deploys production, and neither can reach the other's environment. + - **The release workflow is safe by derivation rather than by a gate, which is the distinction to get right before copying anything from it.** [`publish-release.yml`](./.github/workflows/publish-release.yml) has exactly one **ref** guard, an `Assert dispatch ref step` refusing any ref but `main` or `develop`. It also runs the reusable `validate` job that the pull request runs, so a dispatch cannot release a ref that fails validation, but that gates the *sources* rather than the branch-to-classification relationship this item is about. Everything else follows from the ref with nothing to disagree with it: NBGV reads `publicReleaseRefSpec` in [`version.json`](./version.json) to give `main` a clean `X.Y.Z` and every other branch a `-g` suffix, and the release's `prerelease:` is computed as `github.ref_name != 'main'`. **`main` cannot cut a prerelease because no input exists that could ask it to.** This repo has no `validate-release` job — that is a build-repo construct in the fleet contract, not something shipped here, and an earlier draft of this item claimed it did. + - **The deploy differs in kind, which is why it needs the gate the release does not.** `environment` is an independent `workflow_dispatch` input, so the ref and the environment are two values that can disagree, and today only one of the two disagreements is caught. That is the whole argument: not that the deploy should imitate the release, but that a workflow taking the target as a separate input has to check it, where one deriving the target from the ref has nothing to check. + - **This reverses a deliberate decision, so the comment above the job has to change with the code.** It currently reads "Staging deploys from any ref, since proving a branch before it merges is what staging is for", which is the argument for the present behavior and would be left contradicting the gate. Whoever makes the change states the new reason instead: staging is the rehearsal for what `develop` will promote, so a deploy from a feature branch proves something that is not what ships. Requested by the maintainer 2026-08-09, after a staging deploy was dispatched from `main` here. + - Keep the existing comparison style, against the full `github.ref` rather than `ref_name`, for the reason already recorded there: a tag and a branch can share a short name, so a tag called `develop` would satisfy a `ref_name` test while pointing at an arbitrary commit. The new arm inherits that hazard exactly and is not a special case. - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. +- **[`checks/README.md`](./checks/README.md) names one of the three gates in its own directory.** It is written as the URL contract's document, titled `URL Parity Gate`, and `check-env-docs.py` and `check-eol-pins.py` appear in it nowhere. Each carries its reasoning in its own docstring, which is where the detail belongs, but [#66][pr-66] made a directory's README the authority on that directory and this one currently describes a subset of it without saying so. The small fix is a section naming what else lives there and what each gate answers; the alternative is to state in the README that it is scoped to the contract alone. Either resolves it, and the present state is the one that reads as complete while not being. - Decommission WordPress.com only after **30 clean days**, and downgrade to free rather than deleting, which keeps the media reachable as a safety net and preserves the ability to re-export. Do not start sooner: the conversion fetched media over HTTP from the live site. ## Recurring operations @@ -65,7 +78,9 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y ## Owed to the hub -Nothing. The spec update this repo owed the hub has landed: [ProjectTemplate#560][hub-type-pr] authored the `hugo` type, the `self-hosted` target, the `deploy-ssh` mechanism, guarantees D4.6 and D5.6, and a reference leaf pair, all measured from what this repo actually runs rather than from the prediction the intake carried. [#456][hub-issue] and [#558][hub-spec-issue] are closed with it. +**One thing, and it is conditional.** [ProjectTemplate#633][issue-633] reports that the hub's `repo_gate.py --check eol` compares `.gitattributes` and `.editorconfig` to each other and never to the tree, and offers to port this repo's [`checks/check-eol-pins.py`](./checks/check-eol-pins.py) into the hub's `scripts/` with tests in the `test_repo_gate.py` style. That offer is owed only if the hub wants that shape. **Do not start it before the hub rules**, because the open design question changes the code: the dead-pattern direction cannot be lifted unmodified, since a carried baseline forward-declares pins for files its consumers will have and the template itself does not, which is correct there and a finding anywhere else. + +The spec update this repo owed the hub has landed: [ProjectTemplate#560][hub-type-pr] authored the `hugo` type, the `self-hosted` target, the `deploy-ssh` mechanism, guarantees D4.6 and D5.6, and a reference leaf pair, all measured from what this repo actually runs rather than from the prediction the intake carried. [#456][hub-issue] and [#558][hub-spec-issue] are closed with it. **It is on the hub's `develop` and not on `main`, so it is not ground truth yet.** The registry entry that reclassifies this repo to `types: ["hugo", "source-only"]` with both publish targets sits on the same unpromoted branch. Until the hub promotes, this repo stays `source-only` for audit purposes, and the anticipatory evaluation of the nine `hugo` checks is in [the audit report](./reports/Blog/audit.md). @@ -109,6 +124,7 @@ Three findings are open at the hub, recorded here rather than only in the issues | [#550][issue-550] | Nothing detects a repo missing from the registry, which is how this repo stayed invisible. Three other repos are still absent. | | [#552][issue-552] | The audit flags any carried `AGENTS.md` naming the template repo, and the byte-locked `Fleet Bootstrap` section names it. Carrying the canonical correctly cannot pass, and it is the one finding the current run cannot clear. | | [#597][issue-597] | Filed from here, after a mandatory pre-merge gate in `OPERATIONS.md` was skipped on [#40][pr-40]. The ruling is that a verification a runner cannot perform needs a declared destination, not a better per-repo pointer. Answered by [#598][pr-598], which this repo owes work against once it is ground truth. | +| [#633][issue-633] | Filed from here, 2026-08-09. The hub's `eol` gate is document-to-document, so it reported `0 issue(s)` against the exact tree carrying both defects [#69][pr-69] fixed. Where a repo sets `[*] end_of_line = lf`, as this one does, that gate is vacuously satisfied by any pin at all, including one naming a file that does not exist. Open on the hub's side; the conditional work it may ask of this repo is under **Owed to the hub**. | **What [#598][pr-598] will ask of this repo, once it is ground truth.** It is merged to the hub's `develop` and not to `main`, so it binds nothing yet, per the trap below about reading `main` as ground truth. It makes `Local Verification` a sixth declared `OPERATIONS.md` heading, leading the file as the only pre-merge one. This repo's `OPERATIONS.md` carries **13 level-two headings and matches none of the five declared today**, so the work is a rename and reorder rather than new prose: `Local Verification Before a Pull Request` becomes `Local Verification`, and `Backup and Restore` is one word from the declared `Backup and Recovery`. Both are near-misses rather than absences, which is the shape a heading check will mostly find in a repo that wrote its operational document before the spec declared headings. Do not start until the hub promotes it. @@ -147,6 +163,9 @@ Each of these was hit or nearly hit, and each is cheap to re-trip. - **The Copilot reviewer's login differs by API, and a wrong-form filter reads as a clean review.** REST reports `copilot-pull-request-reviewer[bot]`, GraphQL omits the suffix. A filter written in the other form matches nothing, and an empty result is indistinguishable from no findings. Assert the filter matched before trusting what it returned. - **A Copilot review hides findings in the review body, where the thread API cannot see them.** The `reviewThreads` query returns line threads only, so a review carrying `Suppressed comments (N)` in a `
` block reports zero unresolved while real findings sit unread. Read the review body itself, not just the threads, before calling a review loop finished. - **`gh pr merge --delete-branch` on a `develop -> main` promotion deletes `develop`.** Use a plain `gh pr merge --merge`. +- **A review that exists on the head is not a finished review.** On [#68][pr-68] `pr_review.py wait` returned after one second with `review_on_head=yes unresolved=0 suppressed=0`, an independent thread poll agreed at zero, and the pull request was reported green. Copilot posted a third finding *after* both reads, and the maintainer found it. The re-request had matched a review still being written, so every query was correct and every query was early. Wait and re-sweep all three surfaces, line threads, issue-level comments and the review bodies, before calling a loop finished. A very short `waited=` is the tell. +- **A rule naming a target that does not exist reads as coverage.** `.gitattributes` pinned `deploy/blog-deploy-shell` and `deploy/authorized_keys`, neither ever tracked in this repo, and the comment above them claimed the extensionless-shebang case was handled. It was not, and the one real instance sat unpinned twenty lines up, invisible precisely because the file looked thorough. [`checks/check-eol-pins.py`](./checks/check-eol-pins.py) gates this file now, in both directions. The generalisation is not gated anywhere: a config that names a path, a job, or a host that is absent will read as protection for as long as nobody checks. +- **A gate is only as good as the matcher underneath it, and a wrong matcher fails quietly.** `check-eol-pins.py` shipped with two defects in one function, `git ls-files` pathspec not sharing gitattributes glob semantics and then a leading `/` being stripped before anchoring was decided. Both were found by review rather than by the gate, and each would have made it report a confident wrong answer. [#70][pr-70] fixed them and it now checks its own matching against `git check-attr` on every run, which is the general move: where a check predicts what a tool will do, compare it against the tool rather than against the documentation. ## Reference @@ -182,6 +201,13 @@ The deploy root is deliberately absent from this table. The rsync destination is [migration-post]: ./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md [pr-30]: https://github.com/ptr727/Blog/pull/30 [pr-40]: https://github.com/ptr727/Blog/pull/40 +[pr-65]: https://github.com/ptr727/Blog/pull/65 +[pr-66]: https://github.com/ptr727/Blog/pull/66 +[pr-68]: https://github.com/ptr727/Blog/pull/68 +[pr-69]: https://github.com/ptr727/Blog/pull/69 +[pr-70]: https://github.com/ptr727/Blog/pull/70 +[pr-75]: https://github.com/ptr727/Blog/pull/75 +[pr-76]: https://github.com/ptr727/Blog/pull/76 @@ -196,5 +222,6 @@ The deploy root is deliberately absent from this table. The rsync destination is [issue-554]: https://github.com/ptr727/ProjectTemplate/issues/554 [issue-563]: https://github.com/ptr727/ProjectTemplate/issues/563 [issue-597]: https://github.com/ptr727/ProjectTemplate/issues/597 +[issue-633]: https://github.com/ptr727/ProjectTemplate/issues/633 [pr-553]: https://github.com/ptr727/ProjectTemplate/pull/553 [pr-598]: https://github.com/ptr727/ProjectTemplate/pull/598 diff --git a/checks/check-env-docs.py b/checks/check-env-docs.py index e2ed0f5..e22891d 100755 --- a/checks/check-env-docs.py +++ b/checks/check-env-docs.py @@ -40,7 +40,14 @@ # Set per invocation rather than stored, so they appear in no template and would otherwise # be invisible to this check. Listed here because the doc has a table for them, and a knob # nobody documented is the same failure as an undocumented file value. -KNOBS = {"ENV_FILE", "REQUIRE_BROTLI", "NO_LINK_DEST", "KEEP_RELEASES", "EXPECT_RELEASE"} +KNOBS = { + "ENV_FILE", + "REQUIRE_BROTLI", + "NO_LINK_DEST", + "KEEP_RELEASES", + "EXPECT_RELEASE", + "CHECK_TAG", +} # Names that look like configuration to the patterns above but are not. # ENVIRONMENT and RELEASE_ID are computed inside the workflow and passed down, and diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index 0bafda1..e37f12d 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -40,12 +40,99 @@ done FAILED="$(mktemp)" CURLERR="$(mktemp)" CURLRC="" -trap 'rm -f "$FAILED" "$CURLERR" ${CURLRC:+"$CURLRC"}' EXIT +CHECKRC="$(mktemp)" +trap 'rm -f "$FAILED" "$CURLERR" "$CHECKRC" ${CURLRC:+"$CURLRC"}' EXIT + +# Every request this script makes announces itself as synthetic, so the server's log can be +# filtered down to real visitors with one clause. Agreed with the host side, whose Traefik +# captures the field and whose own `ci/smoke.sh` already sends `vps/smoke`. +# +# The value carries provenance rather than a boolean, `/`, because "which run +# produced this 404" is then a one-line query against the log. +# +# The run attempt is part of the id deliberately. A re-run of a failed workflow keeps the +# same GITHUB_RUN_ID and gets a new GITHUB_RUN_ATTEMPT, so the id alone would merge a +# retried run into the run it was retrying, which is exactly the case someone reads the log +# to understand. +# +# It is forgeable and it gates nothing. Absence of the header is not proof of a human +# either: a scanner sends no header and neither does a forged request. It must never reach +# auth, rate limiting, robots handling, or caching. +if [ -z "${CHECK_TAG:-}" ]; then + if [ -n "${GITHUB_RUN_ID:-}" ]; then + CHECK_TAG="github/${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT:-1}" + else + CHECK_TAG="proxmox/manual" + fi +fi +# Validated before it is written, because this lands in a curl config file and a curl config +# file is a list of options rather than a list of headers. A value carrying a newline ends the +# header line and starts a new directive, so an override could add an option nobody typed; a +# value carrying a double quote ends the quoted string with the same result. Neither is a +# legal HTTP header value either, so refusing both loses nothing. +# +# The shape is enforced and not merely described, because the whole value of provenance over a +# boolean is that the log can be grouped by source, and `select(.tag | startswith("github/"))` +# is only reliable if every tag actually has a source half. A charset check alone would accept +# `smoke`, `/smoke` and `a/b/c`, each of which reads as conforming and breaks that query. +# Exactly one slash, both halves non-empty, from a deliberately narrow character set. +# +# The range `A-Za-z0-9` is collation-dependent, so the allowlist below is only ASCII-strict +# because `globasciiranges` happens to be on. Set explicitly rather than inherited, since a +# guarantee resting on a build default is not a guarantee. Demonstrated rather than assumed: +# with the option off, under en_US.UTF-8, `aé` and `aÉ` are both ACCEPTED by this pattern, +# and with it on they are rejected. +# Checked, because this script runs under `set -uo pipefail` and not `-e`, so an unsupported +# option would print to stderr, return 1, and be stepped straight over — leaving the +# validation locale-dependent underneath a comment promising it is not. `shopt` returns 1 on +# an unknown option name, which is what makes this testable rather than decorative. +shopt -s globasciiranges || { + echo "FAIL this shell does not support globasciiranges, so the character allowlist below would be locale-dependent" >&2 + exit 2 +} +case "$CHECK_TAG" in +*[!A-Za-z0-9._/-]*) + echo "FAIL CHECK_TAG may contain only letters, digits, and the characters '. _ - /' -- got '$CHECK_TAG'" >&2 + exit 2 + ;; +*/*/*) + echo "FAIL CHECK_TAG takes exactly one slash, as / -- got '$CHECK_TAG'" >&2 + exit 2 + ;; +/* | */) + echo "FAIL CHECK_TAG needs a non-empty half either side of the slash -- got '$CHECK_TAG'" >&2 + exit 2 + ;; +*/*) ;; +*) + echo "FAIL CHECK_TAG must be /, such as proxmox/media-dev -- got '$CHECK_TAG'" >&2 + exit 2 + ;; +esac +printf 'header = "X-Blog-Check: %s"\n' "$CHECK_TAG" >"$CHECKRC" +echo "==> tagging requests X-Blog-Check: $CHECK_TAG" # A resource access token opens the proxy's auth gate. # It goes into a curl config file because bash cannot export an array to the parallel checks. # A command line is also world-readable in ps output, and this runs 1,245 of them. if [ -n "${PANGOLIN_ACCESS_TOKEN_ID:-}" ] && [ -n "${PANGOLIN_ACCESS_TOKEN:-}" ]; then + # Same hazard as CHECK_TAG above and the same reason, but a narrower rule, because the + # grammar of a credential is the issuer's to define and not this script's. Only the + # characters that break out of a quoted config line are refused, and none is legal in an + # HTTP header value, so a token containing one is a paste accident rather than a token. + # Reported without echoing the value, since it is a secret and the finding is its shape. + # + # Carriage return counts as a line ending here as much as newline does. Header injection + # is classically CRLF, and a lone CR is enough on its own, so refusing LF while allowing + # CR would leave the shape this guard exists for. + for name in PANGOLIN_ACCESS_TOKEN_ID PANGOLIN_ACCESS_TOKEN; do + case "${!name}" in + *'"'* | *$'\n'* | *$'\r'*) + echo "FAIL $name contains a quote, a newline, or a carriage return, none of which can appear in an HTTP header value" >&2 + exit 2 + ;; + esac + done CURLRC="$(mktemp)" chmod 600 "$CURLRC" printf 'header = "P-Access-Token-Id: %s"\nheader = "P-Access-Token: %s"\n' \ @@ -58,14 +145,18 @@ elif [ -n "${PANGOLIN_ACCESS_TOKEN_ID:-}" ] || [ -n "${PANGOLIN_ACCESS_TOKEN:-}" fi # Assembled once here rather than per request, since it is the same for every call. -AUTH=() -[ -n "$CURLRC" ] && AUTH=(-K "$CURLRC") +# The check tag is unconditional and the token is not, which is why they are two files +# rather than one. Every request should be attributable; only a same-origin request may +# carry the credential, and folding them together would make the tag inherit that +# restriction for no reason, or the token lose it, depending on which way it was folded. +AUTH=(-K "$CHECKRC") +[ -n "$CURLRC" ] && AUTH+=(-K "$CURLRC") # Invoked indirectly, through `export -f` and the `xargs bash -c` calls below. # shellcheck disable=SC2329 check_render() { - local url="$1" code auth=() - [ -n "$CURLRC" ] && auth=(-K "$CURLRC") + local url="$1" code auth=(-K "$CHECKRC") + [ -n "$CURLRC" ] && auth+=(-K "$CURLRC") code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${auth[@]}" "$BASE$url") [ "$code" = "200" ] || echo "render $url expected 200, got $code" >>"$FAILED" } @@ -81,8 +172,8 @@ check_render() { # arrived truncated to nothing, which still answers 200. Content type is asserted because a # server misconfigured into serving an error page for a missing asset answers 200 as well. check_media() { - local url="$1" code len type target auth=() target_auth=() - [ -n "$CURLRC" ] && auth=(-K "$CURLRC") + local url="$1" code len type target auth=(-K "$CHECKRC") target_auth=() + [ -n "$CURLRC" ] && auth+=(-K "$CURLRC") target="$BASE$url" target_auth=("${auth[@]}") # One hop is followed rather than passed to curl -L, because -L would carry the @@ -102,10 +193,10 @@ check_media() { # Same origin boundary as check_redirect, and for the same reason: a rule that one # day points off-site must not mail the token there. A bare prefix would also accept # a lookalike host registered as an attacker's subdomain. - target_auth=() + target_auth=(-K "$CHECKRC") if [ -n "$CURLRC" ]; then case "$target" in - "$BASE" | "$BASE"/*) target_auth=(-K "$CURLRC") ;; + "$BASE" | "$BASE"/*) target_auth+=(-K "$CURLRC") ;; esac fi ;; @@ -144,8 +235,8 @@ check_media() { # Invoked indirectly, the same way as check_render above. # shellcheck disable=SC2329 check_redirect() { - local url="$1" code dest dcode auth=() dest_auth=() - [ -n "$CURLRC" ] && auth=(-K "$CURLRC") + local url="$1" code dest dcode auth=(-K "$CHECKRC") dest_auth=(-K "$CHECKRC") + [ -n "$CURLRC" ] && auth+=(-K "$CURLRC") code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${auth[@]}" "$BASE$url") case "$code" in 301 | 308) ;; @@ -162,7 +253,7 @@ check_redirect() { # starts with this one, such as a lookalike registered as an attacker's subdomain. if [ -n "$CURLRC" ]; then case "$dest" in - "$BASE" | "$BASE"/*) dest_auth=(-K "$CURLRC") ;; + "$BASE" | "$BASE"/*) dest_auth+=(-K "$CURLRC") ;; esac fi dcode=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${dest_auth[@]}" "$dest") @@ -174,7 +265,7 @@ check_redirect() { } export -f check_render check_redirect check_media -export BASE FAILED CURLRC +export BASE FAILED CURLRC CHECKRC echo "==> $BASE" diff --git a/deploy/make-release.sh b/deploy/make-release.sh index 89c5eec..170471c 100755 --- a/deploy/make-release.sh +++ b/deploy/make-release.sh @@ -19,6 +19,10 @@ usage() { REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +# The release upstream replaced `git whatchanged` with `git log`. Anything older restores +# nothing and exits 0, so this is the floor rather than a preference. +MTIME_MIN=2025.08 + # The deploy root and the base URL are the only host-specific values, and they pair per environment. # ENV_FILE selects the environment, because `set -a` overwrites a value the caller exported. # The first argument overrides the root, being read after this. @@ -82,6 +86,121 @@ command -v hugo >/dev/null || { cd "$REPO" +# Git stores no mtimes, so a checkout stamps every file with the moment it was written, and +# a release built from a fresh clone then links nothing against the previous one. This host's +# long-lived working tree has old mtimes already and links fine, which is exactly what makes +# the gap easy to miss: it is invisible here and total in a clean checkout. +# +# The deploy workflow does the same thing with the same assertion after it, deliberately, so +# the local path and CI fail the same way for the same reason rather than one of them being +# the trusted one. +# +# Required rather than optional. Skipping when absent is how the CI version shipped broken +# for four releases: it printed a reassuring line and restored nothing. +# Both invocation forms are accepted, because how it installs decides which one resolves. The +# Debian and Ubuntu package puts it in git's exec-path at /usr/lib/git-core, where only the +# subcommand form works; a manual install to /usr/local/bin gives the bare name and no +# subcommand. Testing only one would refuse a correctly installed tool. +# +# Each candidate is version-checked and the first ACCEPTABLE one wins, rather than the first +# one that merely exists. A host can carry both, and an old manual install must not veto a +# current packaged one sitting behind it. +# +# The version is gated rather than left to the assertion below, because before MTIME_MIN the +# tool calls `git whatchanged`, which current git refuses to run, so it reports files to be +# processed, processes none, and exits 0. Refusing it here names the cause; the assertion can +# only report the symptom. Versions are YYYY.MM, so dropping the dot compares them as integers. +mtime_probe() { + # The failed match is tolerated because `set -e` with `pipefail` would otherwise abort the + # whole script at the assignment, making every diagnostic below unreachable. + "$@" --version 2>/dev/null | grep -oE '[0-9]{4}\.[0-9]{2}' | head -1 || true +} + +MTIME_CMD=() +mtime_version="" +mtime_found="" +for mtime_form in bare subcommand; do + mtime_try=() + case "$mtime_form" in + bare) command -v git-restore-mtime >/dev/null 2>&1 && mtime_try=(git-restore-mtime) ;; + subcommand) git restore-mtime --version >/dev/null 2>&1 && mtime_try=(git restore-mtime) ;; + esac + [ ${#mtime_try[@]} -gt 0 ] || continue + + mtime_try_version="$(mtime_probe "${mtime_try[@]}")" + if [ -z "$mtime_try_version" ]; then + mtime_found="${mtime_found}${mtime_found:+, }${mtime_try[*]} (no version reported)" + continue + fi + mtime_found="${mtime_found}${mtime_found:+, }${mtime_try[*]} $mtime_try_version" + if [ "${mtime_try_version//./}" -ge "${MTIME_MIN//./}" ]; then + MTIME_CMD=("${mtime_try[@]}") + mtime_version="$mtime_try_version" + break + fi +done + +if [ ${#MTIME_CMD[@]} -eq 0 ]; then + if [ -z "$mtime_found" ]; then + echo "git-restore-mtime not found, as either 'git-restore-mtime' or 'git restore-mtime'" >&2 + else + echo "no usable git-restore-mtime: found $mtime_found, and $MTIME_MIN or newer is required" >&2 + echo " before $MTIME_MIN it calls 'git whatchanged', which current git refuses to run, so it" >&2 + echo " restores nothing and still exits 0 -- every release would silently be a full copy" >&2 + fi + echo " it is what makes --link-dest able to link, and a release built without it is a full copy" >&2 + echo " install git-tools $MTIME_MIN or newer, from https://github.com/MestreLion/git-tools" >&2 + exit 1 +fi + +echo "==> restoring file mtimes with ${MTIME_CMD[*]} $mtime_version" +"${MTIME_CMD[@]}" static + +# Asserted rather than trusted, because the failure this exists for is a restore that reports +# success and does nothing. A restored file cannot be newer than the commit it was dated from, +# so nothing under static/ may be newer than HEAD's commit time. +# +# Locally modified files are excluded, which is the one way this differs from CI. A working +# tree can legitimately hold a static file newer than any commit; a fresh CI checkout cannot, +# so there the same check needs no exclusion. Comparing the clean files only keeps the +# assertion meaningful during an edit loop instead of being skipped whenever the tree is dirty. +mtime_bound="$(git log -1 --format=%ct)" + +# `git status --porcelain` covers modified, staged and untracked in one list, so an empty +# result means every file under static/ is tracked and unchanged. That is the CI case, and it +# takes the same one-pass `find` the workflow uses. +# A rename or a copy emits TWO NUL records, `XY ` then a bare ``, so the loop has to +# consume the second explicitly. Reading it as another status record would strip three +# characters off a bare path and record `tic/a.txt` for `static/a.txt`, leaving the real path +# unexcluded and the assertion able to fail on a file that is legitimately uncommitted. +# Both halves of a rename are excluded, since both are uncommitted. +declare -A mtime_dirty=() +while IFS= read -r -d '' entry; do + mtime_dirty["${entry:3}"]=1 + case "${entry:0:1}" in + R | C) IFS= read -r -d '' mtime_orig && mtime_dirty["$mtime_orig"]=1 ;; + esac +done < <(git status --porcelain -z -- static) + +if [ ${#mtime_dirty[@]} -eq 0 ]; then + mtime_newest=$(find static -type f -printf '%T@\n' | sort -n | tail -1 | cut -d. -f1) +else + echo "==> ${#mtime_dirty[@]} uncommitted path(s) under static/, excluded from the mtime check" + mtime_newest=0 + while IFS= read -r -d '' f; do + [ -n "${mtime_dirty[$f]:-}" ] && continue + t=$(stat -c %Y "$f") + [ "$t" -gt "$mtime_newest" ] && mtime_newest=$t + done < <(git ls-files -z -- static) +fi + +if [ "$mtime_newest" -gt "$mtime_bound" ]; then + echo "mtime restore did nothing: static/ holds unmodified files newer than HEAD's commit," >&2 + echo " so they still carry their checkout time and --link-dest will link nothing" >&2 + exit 1 +fi +echo "==> mtimes restored, newest $mtime_newest against HEAD $mtime_bound" + # Hugo maps HUGO_ onto config, so HUGO_BASEURL overrides hugo.yaml with no flag. # A mirror built without it serves canonical tags, feed links, and permalinks pointing at production. # The build gate passes either way, so the effective value is logged rather than left implicit.