From c5acdd2db192673afc4a125bad7b2e8a736ce306 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 08:57:45 -0700 Subject: [PATCH 01/11] Address two environments instead of one The release tooling could name a single deploy root, which was enough while there was one mirror and will not be once the VPS grows a staging site and a production site. Nothing here could address two at a time. The blocker was not the obvious one. make-release.sh sourced secrets/.env with `set -a`, which exports every assignment in the file and overwrites a DEPLOY_ROOT the caller exported first, so sourcing a different environment file silently deployed to the original root. On a host serving two sites that failure is not an error, it publishes to the other site. ENV_FILE now selects the file, the first argument still overrides the root because it is read afterwards, and a named file that does not exist is a hard failure rather than a fall-through to whatever the ambient environment holds. check-live-urls.sh could not authenticate, so it could not check a site that keeps its auth gate on. Staging keeps its gate on deliberately: it serves a byte-identical copy of the public site, and an open one is a duplicate handed to every crawler, which is the one thing a migration about preserving URLs should not do. A Pangolin resource access token opens it instead. The credential travels in a mode-600 curl config file rather than in -H arguments. A command line is readable in ps for the life of the process and this runs 1,245 of them, and the config file is also the only form that survives the `export -f` the parallel checks run under, since bash cannot export an array. It is sent to the base URL's own origin and nowhere else, so a rule that one day redirects off-site cannot carry it away. A preflight request runs first, because behind an auth gate a wrong token fails every URL and the output then reads as a site that has vanished rather than as a bad credential. Two decisions the docs asserted the opposite of, both now recorded with their reasoning rather than silently reversed. One deploy key covers both environments instead of one per environment: the split pays off only where the two keys never share a machine, and both sit on one workstation and in one secret store. The cost is that the rrsync forced command can no longer separate the environments, so the deploy roots move under a common parent and one pinned command covers both. And staging takes the VPS wildcard at blog.vps.insanegenius.net, which needs no new certificate and closes the open FQDN decision. Verified against both local mirrors, which are two containers in a sibling repository's new blog stack: 1,245 URLs honored on each, each baking its own base URL, neither disturbing the other's release history. Each new gate was demonstrated failing before being trusted, per the rule in TODO.md. Co-Authored-By: Claude Opus 5 (1M context) --- OPERATIONS.md | 68 +++++++++++++++++++++++++++++++++------ README.md | 44 ++++++++++++------------- TODO.md | 11 ++++--- checks/check-live-urls.sh | 58 ++++++++++++++++++++++++++++----- deploy/README.md | 37 ++++++++++++++++++--- deploy/env.example | 33 +++++++++++++++++-- deploy/make-release.sh | 23 ++++++++++--- 7 files changed, 219 insertions(+), 55 deletions(-) diff --git a/OPERATIONS.md b/OPERATIONS.md index ce954c0..c270a97 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -4,13 +4,22 @@ How this site is built, released, served, and rolled back. [`GOVERNANCE.md`](./G ## Environments +Four environments, in two pairs. Each pair is one publish site and one staging site, and the local pair exists to rehearse the remote one. + | Environment | Address | Fronted by | Purpose | | --- | --- | --- | --- | -| Local mirror | a private hostname, set in `secrets/.env` | a reverse proxy on the maintainer's own network | Proves the artifact. The redirect rules, the maps, and the release mechanics. | -| Staging | on the VPS, behind the auth gate | Pangolin | Proves the infrastructure. Routing, TLS, and the deploy path. | +| Local publish mirror | a private hostname, set in `secrets/.env` | Traefik, on the maintainer's own network | Proves the artifact. The redirect rules, the maps, and the release mechanics. | +| Local staging mirror | a second private hostname, set in `secrets/staging.env` | Traefik | Proves that two environments on one host stay independent, before that matters on a server. | +| Staging | `blog.vps.insanegenius.net`, behind the auth gate | Pangolin | Proves the infrastructure. Routing, TLS, and the deploy path. | | Production | `blog.insanegenius.com` | Pangolin | The public site. | -The local mirror is not staging. It runs the same bundle against the same web server, so it catches a broken redirect or a bad permission for free, but it exercises none of the routing, authentication, or certificate machinery that only exists on the VPS. Passing locally says the artifact is right. It says nothing about whether the server in front of it is. +The local mirrors are not staging. They run the same bundle against the same web server, so they catch a broken redirect or a bad permission for free, but they exercise none of the routing, authentication, or certificate machinery that only exists on the VPS. Passing locally says the artifact is right. It says nothing about whether the server in front of it is. + +Each environment is one file under `secrets/`, selected with `ENV_FILE`, holding the deploy root, the base URL, and the container to restart. Selecting the file is how an environment is chosen: the file is sourced with `set -a`, so it overwrites a `DEPLOY_ROOT` the caller exported and setting that variable by hand does not switch anything. A named file that does not exist is a hard failure rather than a fall-through, because on a host serving two sites the ambient value is the other site's root. + +**The staging FQDN sits under the VPS wildcard deliberately.** `blog.vps.insanegenius.net` needs no new certificate and no new DNS record, and it keeps the staging name off the production domain. + +**Staging keeps its auth gate on.** It serves a byte-identical copy of the public site, so exposing it publicly would hand every crawler a duplicate of a site whose entire migration risk is URL preservation. `checks/check-live-urls.sh` gets through with a Pangolin resource access token instead. See [Checking a Site Behind the Auth Gate](#checking-a-site-behind-the-auth-gate). ## The Release Bundle @@ -55,13 +64,24 @@ docker restart "$CADDY_CONTAINER" # required: see below checks/check-live-urls.sh "$HUGO_BASEURL" ``` +Against the staging mirror, name its file in both places, since the sourced values and the ones `make-release.sh` reads must describe the same environment: + +```sh +set -a; . secrets/staging.env; set +a +ENV_FILE=secrets/staging.env deploy/make-release.sh +docker restart "$CADDY_CONTAINER" +checks/check-live-urls.sh "$HUGO_BASEURL" +``` + **Restart every time, even though only some changes strictly need it.** Caddy expands `import` at config-parse time, both for the site config and for the `map` blocks that read `maps/*.map`, and it does not watch those files. Swapping the `current` symlink therefore changes what a *static file* request resolves to, per request, while the redirect rules and map tables stay exactly as they were when Caddy last loaded. Verified against the running mirror: a new map entry present in the live release on disk returned 404 until the container was restarted, then 301. So the failure is specific. **When the release changed `deploy/Caddyfile` or anything under `deploy/maps/`**, checking without a restart exercises the **previous** rules, and a broken redirect reports `PASS` while the shipped artifact is broken. A content-only release does not have this problem, because the rules Caddy already holds are still the right ones. The step is unconditional anyway, for two reasons. Deciding correctly means knowing whether anything reached the config, which is easy to get wrong when a change spans several paths or a map was regenerated as a side effect. And getting it wrong is silent, since the wrong answer is a green check rather than an error. A restart costs a few seconds on a static site, which is cheaper than reasoning about it each time. -Sourcing `secrets/.env` first puts the deploy root and the base URL in the environment, so no literal value is typed. `make-release.sh` then needs no arguments, because its deploy root falls back to `$DEPLOY_ROOT` and its version falls back to a timestamp. It still accepts both, and [Deploying](#deploying) below passes them explicitly, which is what CI does so a pipeline run names the commit it built rather than the clock. Either form works locally, and the argument wins over the environment. +Sourcing the environment file first puts the deploy root and the base URL in the environment, so no literal value is typed. `make-release.sh` then needs no arguments, because its deploy root falls back to `$DEPLOY_ROOT` and its version falls back to a timestamp. It still accepts both, and [Deploying](#deploying) below passes them explicitly, which is what CI does so a pipeline run names the commit it built rather than the clock. Either form works locally, and the argument wins over the environment. + +`ENV_FILE` is set as well as sourced, and the redundancy is deliberate. The script sources its own file regardless, so leaving `ENV_FILE` off would build and install against `secrets/.env` while the shell's `$CADDY_CONTAINER` and `$HUGO_BASEURL` still named staging, and the run would restart and check the staging site after publishing to the publish root. The script prints the file it read, on every build, for that reason. It refuses to install a release that fails the build gate. `check-live-urls.sh` does take a base URL, which is where the sourced `$HUGO_BASEURL` goes. It follows all 1,245 URLs against the running mirror, checking each redirect's destination rather than trusting its status code. @@ -69,6 +89,28 @@ Expect `PASS - 1245 URLs honored`. Anything less is a finding, and the output na A documentation-only or workflow-only change does not need this. A change to the four paths above does, because for those CI's green is not evidence. +## Checking a Site Behind the Auth Gate + +Staging keeps Pangolin's authentication on, so an unauthenticated request never reaches the site. `check-live-urls.sh` presents a Pangolin resource access token when both halves of the pair are set, and sends nothing when neither is: + +```sh +set -a; . secrets/staging.env; set +a +checks/check-live-urls.sh "$HUGO_BASEURL" +``` + +| Variable | Header | +| --- | --- | +| `PANGOLIN_ACCESS_TOKEN_ID` | `P-Access-Token-Id` | +| `PANGOLIN_ACCESS_TOKEN` | `P-Access-Token` | + +Set both or neither. Half a pair is a typo rather than a choice, and it is rejected as one rather than presented as a failing site. + +Three properties of how the credential is handled, each there for a reason worth keeping: + +- **It travels in a mode-`600` curl config file, not in `-H` arguments.** A command line is readable in `ps` for the life of the process, and this runs 1,245 of them. The config file is also the only form that survives the `export -f` the parallel checks run under, because bash cannot export an array. +- **It is sent to the base URL's own origin and nowhere else.** The check follows every redirect's destination, and every destination in the contract is same-origin today. A rule that one day points off-site must not mail the credential to whoever is on the other end. +- **A preflight request runs before the 1,245.** Behind an auth gate a wrong token fails *every* URL, and the output then reads as a site that has vanished rather than as a bad credential. The two are indistinguishable from the far end of a CI log, so the run stops on the first request with a message naming which of the two it was. + ## Deploying ```sh @@ -76,12 +118,13 @@ HUGO_BASEURL= deploy/make-release.sh "$(git rev-parse -- checks/check-live-urls.sh ``` -The deploy root and the base URL are the only host-specific values. A local run reads them from an untracked `secrets/.env`, copied from [`deploy/env.example`](./deploy/env.example), and CI passes both explicitly. The whole `secrets/` directory is gitignored, so no address, path, or container name belonging to one machine reaches the published history. +The deploy root and the base URL are the only host-specific values. A local run reads them from an untracked file under `secrets/`, one per environment, copied from [`deploy/env.example`](./deploy/env.example), and CI passes both explicitly. The whole `secrets/` directory is gitignored, so no address, path, or container name belonging to one machine reaches the published history. **Always set `HUGO_BASEURL` for anything that is not production.** The base URL is baked into the canonical tag, the feed links, and every absolute permalink, so a mirror built without it serves pages that all point back at the production address. Nothing downstream catches this, because the pages render at the right paths and the build gate passes. The effective value is printed on every build for that reason. | Variable | Effect | | --- | --- | +| `ENV_FILE` | Which environment file to source. Defaults to `secrets/.env`. | | `DEPLOY_ROOT` | Fallback deploy root. The first argument wins. | | `HUGO_BASEURL` | Overrides the site base URL. | | `REQUIRE_BROTLI=1` | Fails rather than shipping gzip-only. CI sets this. | @@ -119,6 +162,8 @@ The container mounts the deploy root **read-only**, and mounts the **parent** ra Routing differs by environment and the bundle does not. Traefik on the home host has the Docker provider enabled, so container labels route. Pangolin's Traefik on the VPS does not, so routing there is created in the Pangolin UI and labels are silently ignored. +**Each environment is its own container with its own deploy root**, rather than one server addressing several roots. That is what keeps the bundle's config internal: the Caddyfile inside a release names `/srv/blog/current`, one root, and knows nothing about a sibling. A single server covering both would have to name both roots in a config held outside either bundle, and that config could not then roll back with the content it serves. Both containers bind the same internal port and are told apart by hostname, which the proxy in front resolves. + ### The bootstrap, and why it is not in the release The container reads three host paths, and only one of them a release ever writes: @@ -131,17 +176,19 @@ The container reads three host paths, and only one of them a release ever writes [`deploy/bootstrap.Caddyfile`](./deploy/bootstrap.Caddyfile) goes in the `config` directory and is the **only** Caddy file outside the release bundle. It carries a single `import` and no rules of its own, deliberately: everything describing the site ships inside the release, so a rollback reverts the rules and the content together. Rules held here instead would leave a rolled-back site being served by the current release's redirects. -Because it sits outside the bundle, no release updates it. Install or refresh it explicitly: +Because it sits outside the bundle, no release updates it. Install or refresh it explicitly, once per environment, which is the same command against a different sourced file: ```sh -set -a; . secrets/.env; set +a +set -a; . secrets/.env; set +a # or secrets/staging.env install -m 644 deploy/bootstrap.Caddyfile "$CADDY_APPDATA/config/Caddyfile" docker restart "$CADDY_CONTAINER" ``` +**A container started before its environment has a release restart-loops**, because the bootstrap imports a path that does not exist yet. Create the directories, install the bootstrap, cut the first release, and start the container in that order. The container definition can also be held disabled until the release exists, which is the same fix from the other side. + A restart is needed whenever **any** Caddy config changes, not only this file. That includes `deploy/Caddyfile` and anything under `deploy/maps/`, because both are expanded at config-parse time and Caddy does not watch them. Only static file requests follow the `current` symlink per request. See "Local Verification Before a Pull Request" above, where skipping the restart is the difference between a real check and a false pass. -`CADDY_APPDATA` is recorded in `secrets/.env` for exactly this reason. No script reads it, so a rebuild would otherwise depend on someone remembering where the bootstrap goes. +`CADDY_APPDATA` is recorded in each environment's file for exactly this reason. No script reads it, so a rebuild would otherwise depend on someone remembering where the bootstrap goes. ## Redirects @@ -168,9 +215,12 @@ The deploy account exists to receive a release and nothing else. - The account is unprivileged and owns only the deploy root. - Its key is restricted in `authorized_keys` with `restrict` and a forced command, so it cannot open a shell, allocate a terminal, or forward a port. -- Each environment has its own key, so a staging deploy cannot reach production. +- **One key covers both environments**, rather than one per environment. Recorded here as a decision rather than an omission, because the opposite is the obvious default and this file asserted it until the two environments actually existed. A per-environment split pays off only where the two keys never share a machine, and here they would: both private keys sit on the maintainer's one workstation, and both secrets in one GitHub store, so whatever reaches one reaches the other. The split would buy a boundary that is already crossed everywhere it is held. +- **The forced command is therefore the only boundary left, and it is confined to the parent of both roots.** That is what a single key costs: `rrsync` pins a key to one directory, so the two deploy roots sit under one parent and one pinned command covers both. A key confined to `/srv/blog` can write either site, which is exactly what having one key means. - Unattended upgrades run with automatic reboot, which is safe because the site is static and the swap survives a restart. +A deploy key that can write a release can already rewrite the site's Caddy config, because [`deploy/Caddyfile`](./deploy/Caddyfile) ships inside the bundle and the bootstrap imports it. Withholding the container's `/config` directory from the same key therefore protects nothing, which is why the bootstrap stays outside the deploy path for the reason given below and not for a security one. + ## Backup and Restore **The deploy root needs no backup.** The site is reproducible from this repository by running the deploy again, so the only thing worth protecting on the server is its configuration: the container definition, the proxy configuration, the deploy account and its restricted key, and the upgrade schedule. diff --git a/README.md b/README.md index 6a63ab8..883a44b 100644 --- a/README.md +++ b/README.md @@ -2,23 +2,22 @@ Pieter Viljoen's blog, and the tooling that builds, verifies, and deploys it. +The live blog is hosted at [blog.insanegenius.com][blog-link]. + ## Build and Distribution -- **Source Code**: [GitHub][blog-link], holding the source, issues, discussions, and CI/CD pipelines. +- **Source Code**: [GitHub][github-link], holding the source, issues, discussions, and CI/CD pipelines. - **Versioned Releases**: [GitHub Releases][releases-link], version-tagged source archives. -The site itself is not distributed as a package. It is built from this source and deployed to a host over SSH, so a release here is a tagged snapshot of the source rather than an artifact to install. - ### Build Status -[![Releases Build][releases-build-shield]][actions-link]\ -[![Last Commit][last-commit-shield]][commits-link]\ -[![License][license-shield]][license] +[![Release Status][release-status-shield]][actions-link]\ +[![Last Commit][last-commit-shield]][commits-link] ### Releases -[![GitHub Release][github-release-shield]][releases-link]\ -[![GitHub Pre-Release][github-pre-release-shield]][releases-link] +[![GitHub Release][release-version-shield]][releases-link]\ +[![GitHub Pre-Release][pre-release-version-shield]][releases-link] ### Release Notes @@ -30,7 +29,7 @@ The site itself is not distributed as a package. It is built from this source an - The URL contract is committed ground truth and gated in CI: 328 addresses that must render, 917 that must redirect, and 778 legacy image URLs that must resolve. - The site is not yet serving its public address. This release is the source and its pipeline, not the cutover. -See [Release History][history] for the full history. +See [Release History][history] for complete release notes and older versions. ## Table of Contents @@ -38,8 +37,6 @@ See [Release History][history] for the full history. - [Configuration](#configuration) - [Questions or Issues](#questions-or-issues) - [Development Environment Setup](#development-environment-setup) -- [3rd Party Tools](#3rd-party-tools) -- [License](#license) ## Use Cases @@ -69,7 +66,7 @@ Deploy paths, environment variables, and the server layout are documented in [OP To discuss a post, use [Discussions][discussions-link]. The site itself carries no comment system, deliberately: comments on the old platform were closed years ago, and a static site has nowhere to put them without adding a third-party service that outlives its usefulness. Discussions gives a reader somewhere to respond without the site taking on a moving part. -For a defect in the site or the tooling, such as a broken link, a missing redirect, or a page that renders wrongly, open an issue on [the repository][blog-link]. +For a defect in the site or the tooling, such as a broken link, a missing redirect, or a page that renders wrongly, open an [Issue][issues-link]. ## Development Environment Setup @@ -91,9 +88,7 @@ checks/check-live-urls.sh "$HUGO_BASEURL" The deploy root and the base URL come from an untracked `secrets/.env`, copied from [deploy/env.example][env-example]. The whole `secrets/` directory is gitignored, so host-specific values stay out of the published history. -Commits are signed. A greenfield repository signs from its first commit, because the branch ruleset rejects unsigned history and re-signing it afterwards needs a force push the ruleset also blocks. - -## 3rd Party Tools +## 3rd Party Tools | Tool | Role | License | | --- | --- | --- | @@ -101,17 +96,18 @@ Commits are signed. A greenfield repository signs from its first commit, because | [PaperMod][papermod-link] | theme, vendored under `themes/` | MIT | | [Caddy][caddy-link] | web server, serving the built site and the redirects | Apache-2.0 | -## License +## License -See [LICENSE][license]. +Licensed under the [MIT License][license]\ +![GitHub License][license-shield] -[github-pre-release-shield]: https://img.shields.io/github/v/release/ptr727/Blog?include_prereleases&label=GitHub%20Pre-Release&logo=github -[github-release-shield]: https://img.shields.io/github/v/release/ptr727/Blog?logo=github&label=GitHub%20Release +[pre-release-version-shield]: https://img.shields.io/github/v/release/ptr727/Blog?include_prereleases&label=GitHub%20Pre-Release&logo=github +[release-version-shield]: https://img.shields.io/github/v/release/ptr727/Blog?logo=github&label=GitHub%20Release [last-commit-shield]: https://img.shields.io/github/last-commit/ptr727/Blog?logo=github&label=Last%20Commit [license-shield]: https://img.shields.io/github/license/ptr727/Blog?label=License -[releases-build-shield]: https://img.shields.io/github/actions/workflow/status/ptr727/Blog/publish-release.yml?event=workflow_dispatch&logo=github&label=Releases%20Build +[release-status-shield]: https://img.shields.io/github/actions/workflow/status/ptr727/Blog/publish-release.yml?event=workflow_dispatch&logo=github&label=Release%20Status @@ -119,12 +115,13 @@ See [LICENSE][license]. -[blog-link]: https://github.com/ptr727/Blog +[github-link]: https://github.com/ptr727/Blog [checks]: ./checks/ [commits-link]: https://github.com/ptr727/Blog/commits [deploy]: ./deploy/ [deploy-readme]: ./deploy/README.md [discussions-link]: https://github.com/ptr727/Blog/discussions +[issues-link]: https://github.com/ptr727/Blog/issues [env-example]: ./deploy/env.example [history]: ./HISTORY.md [hugo-config]: ./hugo.yaml @@ -134,6 +131,7 @@ See [LICENSE][license]. -[caddy-link]: https://caddyserver.com/ -[hugo-link]: https://gohugo.io/ +[blog-link]: https://blog.insanegenius.com +[caddy-link]: https://caddyserver.com +[hugo-link]: https://gohugo.io [papermod-link]: https://github.com/adityatelange/hugo-PaperMod diff --git a/TODO.md b/TODO.md index acd923d..17d30b5 100644 --- a/TODO.md +++ b/TODO.md @@ -10,7 +10,7 @@ The site is built and gated in CI. It is on GitHub, and it is not yet serving it | --- | --- | | Content and media | done. 514 pages, 778 media files hash-verified against the export tar | | URL contract | done. 328 render, 917 redirect, 778 legacy image URLs, all gated | -| Deploy shape | done and proven against a running Caddy, on a local mirror | +| Deploy shape | done and proven against a running Caddy, on a local publish mirror and a local staging mirror | | CI workflows | green. Validation runs on every pull request and feeds the required check | | GitHub repo | public, both rulesets active, `configure.sh check` exits 0 | | Release pipeline | proven end to end. Release `1.0.11` carries the tag, source archive, README, and LICENSE | @@ -25,8 +25,8 @@ The site is built and gated in CI. It is on GitHub, and it is not yet serving it ## Next, in dependency order - Provision the VPS: an unprivileged `blogdeploy` user, the deploy root, and `unattended-upgrades` with automatic reboot. -- Restrict the deploy key with `restrict,command=...`, no pty and no forwarding, so it can do nothing but rsync into `releases/` and swap the symlink. Generate per-environment keys so staging cannot reach production. -- Choose the staging FQDN, add its DNS record, and expose it through Pangolin as a public resource with **no auth**, since CI's live-URL check has to reach it. Authentication defaults to on for a public resource and has to be turned off deliberately. +- Move the two deploy roots under one parent, `/srv/blog/{production,staging}`, and restrict the **single** deploy key with `restrict,command=...`, no pty and no forwarding, pinned to that parent. One key rather than one per environment is a deliberate decision, recorded with its reasoning in [OPERATIONS.md](./OPERATIONS.md#server-hardening): the split only pays where the two keys never share a machine, and both sit on one workstation and in one secret store. The cost is that the forced command can no longer separate the environments, which is why the roots share a parent. +- Add the staging DNS record for `blog.vps.insanegenius.net` and expose it through Pangolin. It sits under the existing VPS wildcard, so no new certificate is needed, and **authentication stays on**: staging serves a byte-identical copy of the public site, and an open one is a duplicate handed to every crawler. `check-live-urls.sh` gets through with a resource access token instead. - Write `deploy-site.yml` and prove it: a dry run that mutates nothing, then a real run, then a forced mid-deploy failure to confirm rollback keeps the site up. Report the measured deploy shape back to [ProjectTemplate#456][hub-issue], which is waiting on it before the publish type can be defined. - Deploy to a temporary production FQDN and validate there before touching the live record. Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. - Watch server logs for 404s daily for the first week, because real traffic finds what the golden list missed. Append anything new to `checks/golden-urls.txt` and add a redirect. @@ -35,7 +35,6 @@ The site is built and gated in CI. It is on GitHub, and it is not yet serving it ## Open decisions -- The staging FQDN name. - `/robots.txt/` and `/osd.xml/` currently sit in `slugs.map` pointing at `/`. The first would be better pointing at the real `/robots.txt`. ## Deliberate deviations from the fleet baseline @@ -71,6 +70,7 @@ Each of these was hit or nearly hit, and each is cheap to re-trip. - **A Picasa URL ending in `-h` serves an HTML wrapper, not an image, with a 200 status.** Check magic bytes rather than status codes when fetching any binary. - **PaperMod uses APIs Hugo deprecated in 0.158**, so `--panicOnWarning` fails on the theme rather than on content. The two overrides in `layouts/` exist to keep that flag on, and they are the reason the flag is a real gate. - **A hard link keeps its inode's mode and ownership**, so `--chmod` and `--no-g` govern only newly transferred files. A badly moded file rides the link chain into every later release. `NO_LINK_DEST=1` mints fresh inodes. +- **`DEPLOY_ROOT=... deploy/make-release.sh` does not select an environment.** The script sources its environment file with `set -a`, which exports every assignment in it and overwrites whatever the caller exported first, so the variable is set and then silently replaced. `ENV_FILE` selects the file, and the first argument overrides the root, because it is read afterwards. With two sites on one host the failure is not an error: it publishes to the other site. A named `ENV_FILE` that does not exist is a hard failure for the same reason. - **`content/` is an imported archive.** Prose, spelling, and style sweeps do not reach it, and `cspell.json` ignores it deliberately. - **A gate is trusted only after it has been demonstrated failing.** Every gate here has been. A list-driven check also needs a length floor, or a truncated list passes while checking almost nothing. - **Do not name any workflow `build-*-task.yml`** while the repo declares `source-only`, since `detect` is literally `["no build-*-task.yml"]`. @@ -99,8 +99,11 @@ Secrets and variables, per environment. The App-token pair is repository-scoped | `DEPLOY_SSH_PRIVATE_KEY` | secret | | `DEPLOY_SSH_HOST`, `DEPLOY_SSH_USER`, `DEPLOY_SSH_KNOWN_HOSTS` | variable | | `DEPLOY_ROOT`, `HUGO_BASEURL` | variable | +| `PANGOLIN_ACCESS_TOKEN_ID`, `PANGOLIN_ACCESS_TOKEN` | secret, staging only | | `CODEGEN_APP_CLIENT_ID`, `CODEGEN_APP_PRIVATE_KEY` | secret, both stores | +`DEPLOY_SSH_PRIVATE_KEY` now holds the same key in both environments, per the decision above. The environment split still carries the deploy root, the base URL, and the staging-only token pair, so it is not decorative. + [migration-post]: ./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index 6ae5458..9bf28c0 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -25,21 +25,45 @@ for list in golden-urls.txt redirect-urls.txt; do done FAILED="$(mktemp)" -trap 'rm -f "$FAILED"' EXIT +CURLRC="" +trap 'rm -f "$FAILED" ${CURLRC:+"$CURLRC"}' EXIT + +# Staging sits behind Pangolin's auth gate, which a resource access token opens. +# The token pair goes into a curl config file rather than onto the command line, for two reasons. +# The check functions run under `export -f` and `xargs bash -c`, and bash cannot export an array, +# so a pair of -H arguments has no way to reach them intact. A command line is also world-readable +# in ps output for as long as the process lives, and this runs 1,245 of them. +if [ -n "${PANGOLIN_ACCESS_TOKEN_ID:-}" ] && [ -n "${PANGOLIN_ACCESS_TOKEN:-}" ]; then + CURLRC="$(mktemp)" + chmod 600 "$CURLRC" + printf 'header = "P-Access-Token-Id: %s"\nheader = "P-Access-Token: %s"\n' \ + "$PANGOLIN_ACCESS_TOKEN_ID" "$PANGOLIN_ACCESS_TOKEN" >"$CURLRC" + echo "==> sending a Pangolin access token" +elif [ -n "${PANGOLIN_ACCESS_TOKEN_ID:-}" ] || [ -n "${PANGOLIN_ACCESS_TOKEN:-}" ]; then + # Half a credential is a typo rather than a choice, and it would otherwise fail as an outage. + echo "FAIL set both PANGOLIN_ACCESS_TOKEN_ID and PANGOLIN_ACCESS_TOKEN, or neither" >&2 + exit 2 +fi + +# Assembled once here rather than per request, since it is the same for every call. +AUTH=() +[ -n "$CURLRC" ] && AUTH=(-K "$CURLRC") # Invoked indirectly, through `export -f` and the `xargs bash -c` calls below. # shellcheck disable=SC2329 check_render() { - local url="$1" code - code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$BASE$url") + local url="$1" code auth=() + [ -n "$CURLRC" ] && auth=(-K "$CURLRC") + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${auth[@]}" "$BASE$url") [ "$code" = "200" ] || echo "render $url expected 200, got $code" >>"$FAILED" } # Invoked indirectly, the same way as check_render above. # shellcheck disable=SC2329 check_redirect() { - local url="$1" code dest dcode - code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$BASE$url") + local url="$1" code dest dcode auth=() dest_auth=() + [ -n "$CURLRC" ] && auth=(-K "$CURLRC") + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${auth[@]}" "$BASE$url") case "$code" in 301 | 308) ;; *) @@ -48,8 +72,11 @@ check_redirect() { ;; esac # A redirect to a 404 is a broken redirect, so the destination is followed rather than trusted. - dest=$(curl -s -o /dev/null -w '%{redirect_url}' --max-time 30 "$BASE$url") - dcode=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$dest") + dest=$(curl -s -o /dev/null -w '%{redirect_url}' --max-time 30 "${auth[@]}" "$BASE$url") + # Every destination in the contract is same-origin, and the credential is only ever sent to + # the origin it belongs to. A rule that one day redirects off-site must not mail the token there. + [ -n "$CURLRC" ] && [ "${dest#"$BASE"}" != "$dest" ] && dest_auth=(-K "$CURLRC") + dcode=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${dest_auth[@]}" "$dest") # The media rule lands on an image, and a directory gains a trailing slash, so both answers are accepted. case "$dcode" in 200 | 301 | 308) ;; @@ -58,10 +85,25 @@ check_redirect() { } export -f check_render check_redirect -export BASE FAILED +export BASE FAILED CURLRC echo "==> $BASE" +# One request before the 1,245, because an auth gate turns a bad credential into a total failure. +# Without this the output is 1,245 lines saying the site is gone, when the site is fine and the +# token is wrong, and the two are indistinguishable from the far end of a CI log. +preflight=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${AUTH[@]}" "$BASE/") +if [ "$preflight" != "200" ]; then + echo "FAIL preflight: $BASE/ answered $preflight, expected 200" >&2 + if [ -n "$CURLRC" ]; then + echo " a token was sent, so check the pair is valid for this resource" >&2 + else + echo " no token was sent. If this site is behind the auth gate, set" >&2 + echo " PANGOLIN_ACCESS_TOKEN_ID and PANGOLIN_ACCESS_TOKEN" >&2 + fi + exit 1 +fi + n_render=$(grep -c . "$CHECKS/golden-urls.txt") echo "==> checking $n_render URLs that must render" grep . "$CHECKS/golden-urls.txt" | xargs -P "$PARALLEL" -I{} bash -c 'check_render "$@"' _ {} diff --git a/deploy/README.md b/deploy/README.md index aa593e6..53b1e85 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -46,15 +46,30 @@ checks/check-live-urls.sh "$HUGO_BASEURL" ``` The deploy root and the base URL are the only host-specific values, and they pair per -environment. Copy [`env.example`](./env.example) to `secrets/.env` and set both. `secrets/` is -gitignored as a whole directory, so a value naming one machine cannot reach a public repo by -being added to a file nobody remembered to ignore. CI passes them explicitly instead, which -keeps a pipeline run self-describing: +environment. Copy [`env.example`](./env.example) to `secrets/.env` and set both. +`secrets/` is gitignored as a whole directory, so a value naming one machine cannot reach a +public repo by being added to a file nobody remembered to ignore. CI passes them explicitly +instead, which keeps a pipeline run self-describing: ```sh HUGO_BASEURL= deploy/make-release.sh "$(git rev-parse --short HEAD)" ``` +**One file per environment, selected by `ENV_FILE`.** `secrets/.env` is the default and is read +when `ENV_FILE` is unset, so a single-environment host needs nothing else: + +```sh +deploy/make-release.sh # secrets/.env +ENV_FILE=secrets/staging.env deploy/make-release.sh # the staging site on the same host +``` + +Selecting the file is the only way to switch environments. The file is sourced with `set -a`, +which exports every assignment in it and **overwrites** a variable the caller exported first, so +`DEPLOY_ROOT=... deploy/make-release.sh` does not do what it looks like. The first argument +still wins, because it is read after the file. A named file that does not exist is a hard +failure rather than a fall-through to the ambient environment, since on a host running two sites +the ambient value is the other site's root. + **Always set `HUGO_BASEURL` for anything that is not production.** The base URL is baked into the canonical tag, the feed links, and every absolute permalink, so a mirror built without it serves pages that all point back at production. Nothing downstream catches this, because the @@ -65,11 +80,25 @@ every build for that reason. | Variable | Effect | | --- | --- | +| `ENV_FILE` | Which environment file to source. Defaults to `secrets/.env`. | | `DEPLOY_ROOT` | Fallback deploy root. The first argument wins. | | `HUGO_BASEURL` | Overrides the site base URL. Hugo maps `HUGO_` onto config natively. | | `REQUIRE_BROTLI=1` | Fails rather than shipping gzip-only. CI sets this. | | `NO_LINK_DEST=1` | Full copy instead of hard-linking from the previous release. | +`checks/check-live-urls.sh` reads two more, and neither reaches `make-release.sh`: + +| Variable | Effect | +| --- | --- | +| `PANGOLIN_ACCESS_TOKEN_ID` | Resource access token id, sent as the `P-Access-Token-Id` header. | +| `PANGOLIN_ACCESS_TOKEN` | The token itself, sent as `P-Access-Token`. | + +Set both or neither; half a pair is rejected as the typo it is. They go to curl through a +mode-`600` config file rather than as `-H` arguments, which keeps the credential out of the +`ps` output of 1,245 requests, and is also the only form that survives the `export -f` the +parallel checks run under. The token is sent to the base URL's own origin and to nothing else, +so a redirect that one day points off-site cannot carry it away. + ## Layout ```text diff --git a/deploy/env.example b/deploy/env.example index 8a8d822..ddab51d 100644 --- a/deploy/env.example +++ b/deploy/env.example @@ -1,11 +1,23 @@ -# Copy to secrets/.env and set for this host. Every value here names a particular machine rather -# than the project, which is why secrets/ is gitignored as a whole directory and this template -# lives outside it. CI sets the deploy values from environment secrets and never reads a file. +# Copy to secrets/.env and set for this host. Every value here names a particular +# machine rather than the project, which is why secrets/ is gitignored as a whole directory and +# this template lives outside it. CI sets the deploy values from environment secrets and never +# reads a file. +# +# One file per environment, selected by ENV_FILE: +# secrets/.env the default, read when ENV_FILE is unset +# secrets/staging.env ENV_FILE=secrets/staging.env deploy/make-release.sh +# +# The file is sourced with `set -a`, so every assignment in it is exported and a variable the +# caller exported first is overwritten. Selecting the file is therefore how an environment is +# chosen, and the first argument to make-release.sh is how its root is overridden. A named file +# that does not exist is a hard failure rather than a fall-through to the ambient environment, +# because on a host running two sites the ambient value is the other site's root. # # Naming convention: the prefix names whatever owns the value, not whatever reads it. # HUGO_ is fixed by Hugo, which maps HUGO_ onto its own config natively. # DEPLOY_ is the release tooling, which writes the deploy root. # CADDY_ is the container, which owns state the release never touches. +# PANGOLIN_ is the proxy, which owns the credential that opens its auth gate. # Written by every release, and mounted read-only by the container at /srv/blog. # make-release.sh also takes it as a first argument, which wins over this value. @@ -30,3 +42,18 @@ CADDY_APPDATA=/path/to/container/appdata # Caddy expands the site config and the map files at parse time and does not watch them, so a # release or a rollback that changes either is not live until the container restarts. CADDY_CONTAINER=blog + +# A Pangolin resource access token, read by check-live-urls.sh, set only for an environment +# behind the auth gate. Staging keeps its gate on: it serves a byte-identical copy of the site, +# and a public one is a duplicate handed to every crawler, which is the one thing a migration +# about preserving URLs should not do. The check punches through with these instead. +# Set both or neither, and leave both unset for a site that is public. +PANGOLIN_ACCESS_TOKEN_ID= +PANGOLIN_ACCESS_TOKEN= + +# Reserved for the remote deploy, which does not exist yet, and read only by CI when it does. +# They are named here so the local file and the GitHub Environment describe the same shape. +# A local run deploys to a path and needs none of them. +#DEPLOY_SSH_HOST= +#DEPLOY_SSH_USER= +#DEPLOY_SSH_KNOWN_HOSTS= diff --git a/deploy/make-release.sh b/deploy/make-release.sh index b01a16d..0a017ec 100755 --- a/deploy/make-release.sh +++ b/deploy/make-release.sh @@ -18,13 +18,28 @@ usage() { REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" # The deploy root and the base URL are the only host-specific values, and they pair per environment. -# CI passes both explicitly, and a local run reads them from an untracked secrets/.env. +# CI passes both explicitly, and a local run reads them from an untracked file under secrets/. # That directory is gitignored whole, so nothing naming this machine reaches a public repo. -if [ -f "$REPO/secrets/.env" ]; then +# +# ENV_FILE selects which environment a local run targets, and sourcing is deliberately not +# conditional on the variable being unset: `set -a` exports every assignment in the file, so a +# DEPLOY_ROOT exported by the caller is overwritten rather than respected. Pointing ENV_FILE at +# the environment's own file is therefore the only way to switch environments, and passing the +# root as the first argument is the only way to override one, because that is read after this. +DEFAULT_ENV_FILE="$REPO/secrets/.env" +ENV_FILE="${ENV_FILE:-$DEFAULT_ENV_FILE}" +if [ -f "$ENV_FILE" ]; then + echo "==> environment: $ENV_FILE" set -a - # shellcheck disable=SC1091 - . "$REPO/secrets/.env" + # shellcheck disable=SC1090,SC1091 + . "$ENV_FILE" set +a +elif [ "$ENV_FILE" != "$DEFAULT_ENV_FILE" ]; then + # The default file is optional, because CI passes every value explicitly and reads no file. + # A file named on the command line is not: a typo there would otherwise fall through to + # whatever the ambient environment holds, which on this host is the other site's root. + echo "environment file not found: $ENV_FILE" >&2 + exit 1 fi ROOT="${1:-${DEPLOY_ROOT:-}}" From 27d08d6e924c433f2ca02dfd34f7c32a22a07f06 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 12:45:44 -0700 Subject: [PATCH 02/11] Deploy to an environment and prove the release is the one answering A release reaches a host by rsync and no restart, so nothing outside the verifier observes whether the rules that answer are the rules shipped. Content follows the current symlink per request and is live immediately, while rules wait on an in-process config reload. A container that never reloads therefore serves new content under a previous release's redirects, and the URL contract passes against a config that was never deployed. The bundle stamps its own version, substituted at install time, and the check compares it before requesting any of the 1,245 URLs. It waits rather than sampling once, because the reload is asynchronous and a check run straight after a deploy races it. The timeout still catches a container that never converges. Caddy also stops watching permanently after one failed config load, without logging that it has given up, so anything that breaks the symlink even briefly ends that container's ability to pick up releases until it is restarted. The failure reports as healthy, which is why the verifier names both causes. Two more headers identify the environment, since every environment serves one bundle on one port and only the container distinguishes them. Robots defaults to the value that is harmless on production, because the failures are asymmetric: an unset staging container is still behind its auth gate, where production inheriting noindex would deindex the site silently. trusted_proxies takes its ranges from the container, and excludes the bridge gateway. Trusting the subnet trusts the host, which was verified by forging a client address from it. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/deploy-site-task.yml | 127 ++++++++++++++++++++++ .github/workflows/deploy-site.yml | 50 +++++++++ OPERATIONS.md | 58 +++++++--- TODO.md | 10 +- checks/check-live-urls.sh | 79 ++++++++++++-- deploy/Caddyfile | 24 +++++ deploy/README.md | 140 +++++++++++++++++++++++-- deploy/env.example | 13 ++- deploy/make-release.sh | 34 ++++-- spec/secrets.json | 9 +- 10 files changed, 498 insertions(+), 46 deletions(-) create mode 100644 .github/workflows/deploy-site-task.yml create mode 100644 .github/workflows/deploy-site.yml diff --git a/.github/workflows/deploy-site-task.yml b/.github/workflows/deploy-site-task.yml new file mode 100644 index 0000000..0ece398 --- /dev/null +++ b/.github/workflows/deploy-site-task.yml @@ -0,0 +1,127 @@ +name: Deploy site task + +on: + workflow_call: + inputs: + environment: + description: The GitHub Environment to deploy to, production or staging. + required: true + type: string + +env: + # Pinned by version and checksum, because the site is reproducible only if the generator is. + # Update both values together. + HUGO_VERSION: 0.164.0 + HUGO_SHA256: 8325f3653032d0fc536503691f4833dc4eb6c6be02ee62466758f3f37a7f2fcd + +jobs: + + # Host-specific values come from the environment, so this file names no host, path, or address. + deploy: + name: Deploy site job + runs-on: ubuntu-latest + environment: ${{ inputs.environment }} + permissions: + contents: read + + steps: + + # Full history, because a shallow clone silently changes page metadata if git info is on. + - name: Checkout code step + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + + - name: Install Hugo step + run: | + set -Eeuo pipefail + deb="hugo_extended_${HUGO_VERSION}_linux-amd64.deb" + curl -sSLf -o "$deb" \ + "https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/${deb}" + echo "${HUGO_SHA256} ${deb}" | sha256sum --check --strict + sudo dpkg --install "$deb" + hugo version + + # REQUIRE_BROTLI below makes a missing binary fatal, so this keeps the build from failing. + - name: Install brotli step + run: | + set -Eeuo pipefail + sudo apt-get update + sudo apt-get install --yes --no-install-recommends brotli + + # Derived once and used three times, as the directory name, the stamp, and EXPECT_RELEASE. + # Deriving it twice yields ids seconds apart, and the gate then asserts a phantom version. + - name: Resolve release id step + id: release + run: | + set -Eeuo pipefail + echo "id=$(date -u +%Y%m%d-%H%M%S)" >> "$GITHUB_OUTPUT" + + # Assembled to a scratch path, since the environment's deploy root is on the far host. + # Naming the root explicitly also marks this a bundle for shipping rather than an install. + - name: Assemble release bundle step + env: + HUGO_BASEURL: ${{ vars.HUGO_BASEURL }} + REQUIRE_BROTLI: '1' + run: | + set -Eeuo pipefail + deploy/make-release.sh "${RUNNER_TEMP}/bundle" "${{ steps.release.outputs.id }}" + + - name: Install deploy key step + env: + DEPLOY_SSH_PRIVATE_KEY: ${{ secrets.DEPLOY_SSH_PRIVATE_KEY }} + DEPLOY_SSH_KNOWN_HOSTS: ${{ vars.DEPLOY_SSH_KNOWN_HOSTS }} + run: | + set -Eeuo pipefail + mkdir -p ~/.ssh + chmod 700 ~/.ssh + printf '%s\n' "$DEPLOY_SSH_PRIVATE_KEY" > ~/.ssh/deploy + chmod 600 ~/.ssh/deploy + printf '%s\n' "$DEPLOY_SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts + chmod 600 ~/.ssh/known_hosts + + # The destination is anchored at the key's confinement root. + # A full host path is remapped beneath that root and fails as an IO error. + # - link-dest points at current, which still resolves to the previous release until the flip. + # - mkpath creates releases/, which does not exist on a fresh environment. + # - delete is omitted, since at an environment root it silently removes rollback targets. + - name: Upload release step + env: + DEPLOY_SSH_USER: ${{ vars.DEPLOY_SSH_USER }} + DEPLOY_SSH_HOST: ${{ vars.DEPLOY_SSH_HOST }} + RELEASE_ID: ${{ steps.release.outputs.id }} + ENVIRONMENT: ${{ inputs.environment }} + run: | + set -Eeuo pipefail + rsync -az --mkpath --chmod=D755,F644 \ + --link-dest="/${ENVIRONMENT}/current/" \ + -e "ssh -i ~/.ssh/deploy -o IdentitiesOnly=yes" \ + "${RUNNER_TEMP}/bundle/releases/${RELEASE_ID}/" \ + "${DEPLOY_SSH_USER}@${DEPLOY_SSH_HOST}:/${ENVIRONMENT}/releases/${RELEASE_ID}/" + + # Separate from the upload, so a failed transfer cannot half-publish a site. + # rsync replaces the symlink through a temporary and a rename, so it is never absent. + - name: Flip current step + env: + DEPLOY_SSH_USER: ${{ vars.DEPLOY_SSH_USER }} + DEPLOY_SSH_HOST: ${{ vars.DEPLOY_SSH_HOST }} + ENVIRONMENT: ${{ inputs.environment }} + run: | + set -Eeuo pipefail + rsync -a --no-recursive \ + -e "ssh -i ~/.ssh/deploy -o IdentitiesOnly=yes" \ + "${RUNNER_TEMP}/bundle/current" \ + "${DEPLOY_SSH_USER}@${DEPLOY_SSH_HOST}:/${ENVIRONMENT}/" + + # The only step that observes the running site. + # An upload succeeds against a container serving nothing, and a flip without a config reload. + # The token pair is set on staging alone, since production answers unauthenticated. + - name: Verify URL contract step + env: + EXPECT_SITE_ENV: ${{ inputs.environment }} + EXPECT_RELEASE: ${{ steps.release.outputs.id }} + PANGOLIN_ACCESS_TOKEN_ID: ${{ secrets.PANGOLIN_ACCESS_TOKEN_ID }} + PANGOLIN_ACCESS_TOKEN: ${{ secrets.PANGOLIN_ACCESS_TOKEN }} + run: | + set -Eeuo pipefail + checks/check-live-urls.sh "${{ vars.HUGO_BASEURL }}" diff --git a/.github/workflows/deploy-site.yml b/.github/workflows/deploy-site.yml new file mode 100644 index 0000000..f6e689c --- /dev/null +++ b/.github/workflows/deploy-site.yml @@ -0,0 +1,50 @@ +name: Deploy site action + +on: + workflow_dispatch: + inputs: + environment: + description: Which environment to deploy. + required: true + type: choice + options: + - staging + - production + +# Runs queue rather than cancel, because a cancelled deploy leaves a release uploaded and unflipped. +concurrency: + group: ${{ github.workflow }}-${{ inputs.environment }} + cancel-in-progress: false + +jobs: + + # The same gate the pull request and a release run. + validate: + name: Validate sources job + uses: ./.github/workflows/validate-task.yml + permissions: + contents: read + + # Staging deploys from any ref, since proving a branch before it merges is what staging is for. + # Asserted before Hugo is installed and before the key reaches the runner. + assert-ref: + name: Assert deploy ref job + runs-on: ubuntu-latest + steps: + - name: Assert ref matches environment step + run: | + set -Eeuo pipefail + if [ "${{ inputs.environment }}" = "production" ] && [ "${{ github.ref_name }}" != "main" ]; then + echo "::error::Deploy production from main; got ${{ github.ref_name }}." + exit 1 + fi + + deploy: + name: Deploy site job + needs: [ validate, assert-ref ] + uses: ./.github/workflows/deploy-site-task.yml + with: + environment: ${{ inputs.environment }} + permissions: + contents: read + secrets: inherit diff --git a/OPERATIONS.md b/OPERATIONS.md index c270a97..0003b31 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -15,7 +15,9 @@ Four environments, in two pairs. Each pair is one publish site and one staging s The local mirrors are not staging. They run the same bundle against the same web server, so they catch a broken redirect or a bad permission for free, but they exercise none of the routing, authentication, or certificate machinery that only exists on the VPS. Passing locally says the artifact is right. It says nothing about whether the server in front of it is. -Each environment is one file under `secrets/`, selected with `ENV_FILE`, holding the deploy root, the base URL, and the container to restart. Selecting the file is how an environment is chosen: the file is sourced with `set -a`, so it overwrites a `DEPLOY_ROOT` the caller exported and setting that variable by hand does not switch anything. A named file that does not exist is a hard failure rather than a fall-through, because on a host serving two sites the ambient value is the other site's root. +**The two words are `production` and `staging`, spelled out, in every position.** No `prod`, no `stage`. The same two name the container, the deploy root, the environment file, the `X-Blog-Env` value, and the GitHub Environment. This is not tidiness: the environment name is a value that gets **compared**, by `EXPECT_SITE_ENV` and by the deploy, so a spelling that differs in one position fails a deploy for a reason that reads like an outage. The local mirrors prefix the same words, `mirror-production` and `mirror-staging`, so a header names exactly one of the four environments in the fleet. + +Each environment is one file under `secrets/`, selected with `ENV_FILE`, holding the deploy root, the base URL, and the container name. Selecting the file is how an environment is chosen: the file is sourced with `set -a`, so it overwrites a `DEPLOY_ROOT` the caller exported and setting that variable by hand does not switch anything. A named file that does not exist is a hard failure rather than a fall-through, because on a host serving two sites the ambient value is the other site's root. **The staging FQDN sits under the VPS wildcard deliberately.** `blog.vps.insanegenius.net` needs no new certificate and no new DNS record, and it keeps the staging name off the production domain. @@ -60,7 +62,6 @@ So release to the local mirror and run the live check **before** opening a pull ```sh set -a; . secrets/.env; set +a deploy/make-release.sh -docker restart "$CADDY_CONTAINER" # required: see below checks/check-live-urls.sh "$HUGO_BASEURL" ``` @@ -69,19 +70,28 @@ Against the staging mirror, name its file in both places, since the sourced valu ```sh set -a; . secrets/staging.env; set +a ENV_FILE=secrets/staging.env deploy/make-release.sh -docker restart "$CADDY_CONTAINER" checks/check-live-urls.sh "$HUGO_BASEURL" ``` -**Restart every time, even though only some changes strictly need it.** Caddy expands `import` at config-parse time, both for the site config and for the `map` blocks that read `maps/*.map`, and it does not watch those files. Swapping the `current` symlink therefore changes what a *static file* request resolves to, per request, while the redirect rules and map tables stay exactly as they were when Caddy last loaded. Verified against the running mirror: a new map entry present in the live release on disk returned 404 until the container was restarted, then 301. +**There is no restart step, and that depends on one flag.** The container runs `caddy run --watch`, which re-adapts the config on a timer and reloads it in process. Re-adapting re-executes every `import`, so a new release's `Caddyfile` and `maps/*.map` are picked up through the unchanged `/config/Caddyfile` that the watcher actually names. Measured on this host: content is live the instant the symlink moves, and the rules follow within about a quarter of a second. + +**The watcher dies silently after one failed config load.** Verified: a flip to a valid release reloads, a flip to a missing one logs the failure and retains the last good config, and a flip back to a valid release **never reloads again**. Nothing in the log says it has given up. Every later deploy then lands content without its rules, which is the failure this section's release stamp exists to catch, and a restart is the only fix. Anything that breaks `current` even briefly, including a test, ends that container's ability to pick up releases. + +**`--watch` needs no admin API**, which is the part worth knowing, because `admin off` makes `caddy reload` impossible and that looks like it should rule out reloading altogether. It does not. `caddy reload` POSTs to the admin endpoint; the watcher reloads in process and never uses it. The log prints `admin endpoint disabled` and `watching config file for changes` together. + +**Without that flag the failure is silent and specific.** Caddy expands `import` at config-parse time and does not watch the imported files, so swapping `current` changes what a *static file* request resolves to, per request, while the redirect rules and map tables stay as they were when Caddy last loaded. Verified both ways against a two-release fixture whose `Caddyfile` was byte-identical and whose map differed: with `--watch`, a flip moved a redirect from 301 to 404 and its replacement from 404 to 301. Without it, neither moved. -So the failure is specific. **When the release changed `deploy/Caddyfile` or anything under `deploy/maps/`**, checking without a restart exercises the **previous** rules, and a broken redirect reports `PASS` while the shipped artifact is broken. A content-only release does not have this problem, because the rules Caddy already holds are still the right ones. +That is why the check verifies the config rather than trusting it. **When the release changed `deploy/Caddyfile` or anything under `deploy/maps/`**, a check run against stale rules exercises the **previous** config, and a broken redirect reports `PASS` while the shipped artifact is broken. The wrong answer is a green check rather than an error, which is the worst shape a failure can take here. -The step is unconditional anyway, for two reasons. Deciding correctly means knowing whether anything reached the config, which is easy to get wrong when a change spans several paths or a map was regenerated as a side effect. And getting it wrong is silent, since the wrong answer is a green check rather than an error. A restart costs a few seconds on a static site, which is cheaper than reasoning about it each time. +So the bundle stamps its own version as `X-Blog-Release`, and `check-live-urls.sh` compares it against `EXPECT_RELEASE` before checking a single URL. It **waits** for a match rather than sampling once, because the reload is asynchronous and a check that starts immediately after a deploy will otherwise race it. The timeout is what still catches a container that is not watching at all, since that one never converges: + +```sh +EXPECT_RELEASE= checks/check-live-urls.sh "$HUGO_BASEURL" +``` Sourcing the environment file first puts the deploy root and the base URL in the environment, so no literal value is typed. `make-release.sh` then needs no arguments, because its deploy root falls back to `$DEPLOY_ROOT` and its version falls back to a timestamp. It still accepts both, and [Deploying](#deploying) below passes them explicitly, which is what CI does so a pipeline run names the commit it built rather than the clock. Either form works locally, and the argument wins over the environment. -`ENV_FILE` is set as well as sourced, and the redundancy is deliberate. The script sources its own file regardless, so leaving `ENV_FILE` off would build and install against `secrets/.env` while the shell's `$CADDY_CONTAINER` and `$HUGO_BASEURL` still named staging, and the run would restart and check the staging site after publishing to the publish root. The script prints the file it read, on every build, for that reason. +`ENV_FILE` is set as well as sourced, and the redundancy is deliberate. The script sources its own file regardless, so leaving `ENV_FILE` off would build and install against `secrets/.env` while the shell's `$HUGO_BASEURL` still named staging, and the run would check the staging site after publishing to the production root. The script prints the file it read, on every build, for that reason. It refuses to install a release that fails the build gate. `check-live-urls.sh` does take a base URL, which is where the sourced `$HUGO_BASEURL` goes. It follows all 1,245 URLs against the running mirror, checking each redirect's destination rather than trusting its status code. @@ -139,12 +149,13 @@ Point `current` at the previous release. The swap is a single rename, so a reque ```sh ln -sfn "releases/" "/.current.tmp" mv -Tf "/.current.tmp" "/current" -docker restart "$CADDY_CONTAINER" ``` -The content reverts on the rename alone, because the container mounts the parent directory and the kernel resolves `current` per request. **The rules do not.** Caddy holds the Caddyfile and the maps as parsed config, so without the restart a rollback serves the previous release's content under the current release's redirects, which is precisely the mismatch that shipping the config inside the bundle exists to prevent. +The content reverts on the rename alone, because the container mounts the parent directory and the kernel resolves `current` per request. The rules follow on the watcher's next poll, within about a quarter of a second, since a rollback is a config change like any other and re-adapting re-reads the reverted release's `Caddyfile` and maps. + +**For that fraction of a second the reverted content is served under the newer release's rules.** That is the same window every deploy has, in the other direction, and it is harmless while every rule is a redirect: a stale redirect sends a visitor to a page that exists in both releases. It would stop being harmless if a rule ever *gated* content rather than redirecting it, and at that point the flip has to become a restart again. -The restart is therefore part of the rollback, not an optional follow-up. It costs a few seconds of downtime on a static site, which is the cheaper half of the trade. +Verify with `EXPECT_RELEASE` set to the release being rolled back **to**, which is what proves the rules actually reverted rather than assuming they did. Verify with `checks/check-live-urls.sh` against the environment before considering the rollback finished. @@ -154,6 +165,20 @@ Ten releases are kept. Unchanged files hard-link to the previous release, so the The script asserts both halves of that rather than assuming them. It fails when the prune leaves more releases than the limit, and when hard-linking produces no shared files at all. Both have failed silently before, and on a compressing filesystem the disk usage looks plausible either way. +## Who Owns What + +The site and the server it runs on are maintained separately, so the boundary is written down rather than inferred. This repo owns the artifact and what proves it correct; the host owns where a release may be written and what happens to it afterwards. + +| This repo | The host | +| --- | --- | +| The GitHub Actions workflow | The SSH endpoint and its forced command | +| `deploy/make-release.sh`, the bundle layout, and the Caddyfile inside it | The bootstrap `import`, the containers, and their environment variables | +| `checks/check-live-urls.sh` and the URL contract | Config-watchdog and release-prune timers | +| The release id and the `@@RELEASE@@` stamp | Proxy resources, routing, tokens, and TLS | +| What a release contains | Where a release may be written, and what happens after | + +The two meet at the container contract in [`deploy/README.md`](./deploy/README.md#container-contract). A defect on the host side is fixed on the host; a pipeline that needs the contract to say something different asks for a contract change rather than growing a second copy of the other side's work. + ## Serving Caddy serves the bundle and binds an internal port only. TLS and the public listener belong to the proxy in front of it, so `auto_https off` and `admin off` are deliberate. @@ -164,6 +189,10 @@ Routing differs by environment and the bundle does not. Traefik on the home host **Each environment is its own container with its own deploy root**, rather than one server addressing several roots. That is what keeps the bundle's config internal: the Caddyfile inside a release names `/srv/blog/current`, one root, and knows nothing about a sibling. A single server covering both would have to name both roots in a config held outside either bundle, and that config could not then roll back with the content it serves. Both containers bind the same internal port and are told apart by hostname, which the proxy in front resolves. +That last sentence is also the risk. **The container is the only thing that distinguishes one environment from another**, so a proxy rule aimed at the wrong one serves the wrong site under the right hostname and returns a healthy `200` with nothing logged anywhere. The bundle stamps `X-Blog-Env` and `X-Robots-Tag` from container variables so a response says which environment produced it, and `checks/check-live-urls.sh` fails on a mismatch when `EXPECT_SITE_ENV` is set. [`deploy/README.md`](./deploy/README.md#identifying-the-environment) carries the mechanism and, more importantly, why `SITE_ROBOTS` defaults to `index, follow` rather than to the safer-looking `noindex`. + +Caddy also sets `trusted_proxies`, because a proxy fronts it in every environment and the peer address is therefore always the proxy. Without it the access log records that one internal address as the client for every request the site serves, and `X-Forwarded-For` is ignored rather than trusted. The ranges come from the container, since the same bundle runs on hosts whose docker subnets differ. **Exclude the bridge gateway from whatever range is trusted**: it is inside the subnet and it is how the host itself reaches the container, so trusting the subnet trusts every process on the host. Verified on the home mirrors by forging a header from the host, which succeeded until the gateway was excluded. **Trusting a range means believing `X-Forwarded-For` from anything in it**, so it is a security boundary rather than a formality, and the bundle's RFC1918 default is only correct where a proxy is genuinely the only thing that can reach the port. [`deploy/README.md`](./deploy/README.md#trusting-the-proxy) carries the three behaviours and why binding to `127.0.0.1` does not make direct access impossible. + ### The bootstrap, and why it is not in the release The container reads three host paths, and only one of them a release ever writes: @@ -181,12 +210,14 @@ Because it sits outside the bundle, no release updates it. Install or refresh it ```sh set -a; . secrets/.env; set +a # or secrets/staging.env install -m 644 deploy/bootstrap.Caddyfile "$CADDY_APPDATA/config/Caddyfile" -docker restart "$CADDY_CONTAINER" +docker restart "$CADDY_CONTAINER" # only this file needs one, see below ``` **A container started before its environment has a release restart-loops**, because the bootstrap imports a path that does not exist yet. Create the directories, install the bootstrap, cut the first release, and start the container in that order. The container definition can also be held disabled until the release exists, which is the same fix from the other side. -A restart is needed whenever **any** Caddy config changes, not only this file. That includes `deploy/Caddyfile` and anything under `deploy/maps/`, because both are expanded at config-parse time and Caddy does not watch them. Only static file requests follow the `current` symlink per request. See "Local Verification Before a Pull Request" above, where skipping the restart is the difference between a real check and a false pass. +**This is the one file whose change still needs a restart**, and the reason is a nice inversion of why everything else does not. The watcher polls `/config/Caddyfile` and reloads when the *adapted result* changes, which is how a release reaches it at all: this file never changes, but re-adapting re-executes its `import` and picks up the new release behind it. Editing this file itself is the case the watcher handles worst, because a bootstrap that no longer parses leaves nothing to reload into. Restart, and read the log. + +Everything inside the bundle, `deploy/Caddyfile` and anything under `deploy/maps/`, reloads without one. See "Local Verification Before a Pull Request" above. `CADDY_APPDATA` is recorded in each environment's file for exactly this reason. No script reads it, so a rebuild would otherwise depend on someone remembering where the bootstrap goes. @@ -216,7 +247,8 @@ The deploy account exists to receive a release and nothing else. - The account is unprivileged and owns only the deploy root. - Its key is restricted in `authorized_keys` with `restrict` and a forced command, so it cannot open a shell, allocate a terminal, or forward a port. - **One key covers both environments**, rather than one per environment. Recorded here as a decision rather than an omission, because the opposite is the obvious default and this file asserted it until the two environments actually existed. A per-environment split pays off only where the two keys never share a machine, and here they would: both private keys sit on the maintainer's one workstation, and both secrets in one GitHub store, so whatever reaches one reaches the other. The split would buy a boundary that is already crossed everywhere it is held. -- **The forced command is therefore the only boundary left, and it is confined to the parent of both roots.** That is what a single key costs: `rrsync` pins a key to one directory, so the two deploy roots sit under one parent and one pinned command covers both. A key confined to `/srv/blog` can write either site, which is exactly what having one key means. +- **The forced command is therefore the only boundary left, and it is confined to the parent of both roots.** That is what a single key costs: `rrsync` pins a key to one directory, so the two deploy roots sit under one parent and one pinned command covers both. The roots are `/srv/blog/sites/production` and `/srv/blog/sites/staging`, and the confinement root is `/srv/blog/sites`. +- **That parent holds content and nothing else, which is why it is not `/srv/blog`.** `/srv/blog` is the deploy account's home directory and contains `/srv/blog/.ssh/authorized_keys`. Confining the key there would let it rewrite the very file that defines what the key may do, and a `--delete` at the root would take `.ssh` with it. Confinement that encloses its own definition is not confinement. The extra `sites/` level is a security boundary rather than tidiness. - Unattended upgrades run with automatic reboot, which is safe because the site is static and the swap survives a restart. A deploy key that can write a release can already rewrite the site's Caddy config, because [`deploy/Caddyfile`](./deploy/Caddyfile) ships inside the bundle and the bootstrap imports it. Withholding the container's `/config` directory from the same key therefore protects nothing, which is why the bootstrap stays outside the deploy path for the reason given below and not for a security one. diff --git a/TODO.md b/TODO.md index 17d30b5..faf9cdc 100644 --- a/TODO.md +++ b/TODO.md @@ -25,7 +25,7 @@ The site is built and gated in CI. It is on GitHub, and it is not yet serving it ## Next, in dependency order - Provision the VPS: an unprivileged `blogdeploy` user, the deploy root, and `unattended-upgrades` with automatic reboot. -- Move the two deploy roots under one parent, `/srv/blog/{production,staging}`, and restrict the **single** deploy key with `restrict,command=...`, no pty and no forwarding, pinned to that parent. One key rather than one per environment is a deliberate decision, recorded with its reasoning in [OPERATIONS.md](./OPERATIONS.md#server-hardening): the split only pays where the two keys never share a machine, and both sit on one workstation and in one secret store. The cost is that the forced command can no longer separate the environments, which is why the roots share a parent. +- The deploy roots are `/srv/blog/sites/{production,staging}` and the confinement root is `/srv/blog/sites`, which is already what the VPS carries. The extra `sites/` level exists because `/srv/blog` is the deploy account's home and holds its own `authorized_keys`, so confining the key there would let it rewrite its own permissions. Restrict the **single** deploy key with `restrict,command=...`, no pty and no forwarding, pinned to that parent. One key rather than one per environment is a deliberate decision, recorded with its reasoning in [OPERATIONS.md](./OPERATIONS.md#server-hardening): the split only pays where the two keys never share a machine, and both sit on one workstation and in one secret store. The cost is that the forced command can no longer separate the environments, which is why the roots share a parent. - Add the staging DNS record for `blog.vps.insanegenius.net` and expose it through Pangolin. It sits under the existing VPS wildcard, so no new certificate is needed, and **authentication stays on**: staging serves a byte-identical copy of the public site, and an open one is a duplicate handed to every crawler. `check-live-urls.sh` gets through with a resource access token instead. - Write `deploy-site.yml` and prove it: a dry run that mutates nothing, then a real run, then a forced mid-deploy failure to confirm rollback keeps the site up. Report the measured deploy shape back to [ProjectTemplate#456][hub-issue], which is waiting on it before the publish type can be defined. - Deploy to a temporary production FQDN and validate there before touching the live record. Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. @@ -69,6 +69,14 @@ Each of these was hit or nearly hit, and each is cheap to re-trip. - **Never populate media over HTTP, and never trust a file count.** WordPress.com serves optimized derivatives at the same URL and filename. Verify by content hash against the export tar. - **A Picasa URL ending in `-h` serves an HTML wrapper, not an image, with a 200 status.** Check magic bytes rather than status codes when fetching any binary. - **PaperMod uses APIs Hugo deprecated in 0.158**, so `--panicOnWarning` fails on the theme rather than on content. The two overrides in `layouts/` exist to keep that flag on, and they are the reason the flag is a real gate. +- **`caddy run --watch` reloads rules and maps with `admin off`, and without it the deploy is silently stale.** `admin off` blocks `caddy reload`, which looks like it rules out reloading, and does not: the watcher reloads in process and never touches the admin endpoint. The watcher names only `/config/Caddyfile`, which never changes, and works anyway because Caddy re-adapts the whole config each poll and re-adapting re-executes the `import`. Drop the flag and the content symlink moves while the rules do not, so the URL check passes against a config that was never deployed. `X-Blog-Release` and `EXPECT_RELEASE` exist to turn that into an error. +- **The reload is asynchronous, so a check run straight after a deploy races it.** Content follows the symlink per request and is live instantly, the rules land about a quarter of a second later. Caught in practice: two environments deployed in one loop, the first passed and the second failed on a header that had not appeared yet. The check waits for `EXPECT_RELEASE` rather than sampling once, and the timeout is what still catches a container that is not watching at all. +- **A bind mount over `/config` shadows the world-writable `caddy/` the image pre-creates there**, so the `--watch` autosave has to `mkdir` and fails on a read-only mount, logging an ERROR once per deploy. `XDG_CONFIG_HOME=/data` moves it. Mount `/config` read-only *and* set that variable, since doing one alone trades a stray `autosave.json` in appdata for a per-deploy error. +- **`caddy run --watch` stops watching permanently after ONE failed config load, and says nothing.** Reproduced: flip `current` to a valid release and it reloads; point it at a missing release and it logs the failure and keeps the last good config; **restore it to a valid release and it never reloads again**. The log's last line is the original failure, then silence, and every other signal says the container is healthy. A restart is the only remedy, and it is the one nobody would try. Consequence for this repo: a deploy after any config-load failure lands content without rules, which `EXPECT_RELEASE` catches but cannot fix. The invariant worth monitoring is that the served `X-Blog-Release` equals the one in `/current/Caddyfile`. +- **A dangling `current` reads as a release mismatch unless you look for it.** Caddy keeps its **last good config** when the imported release vanishes, rather than dying, so `X-Blog-Env` and `X-Blog-Release` stay correct and plausible while every URL 404s underneath them. A monitor checking only the environment header reports the site healthy. The preflight distinguishes the two: a non-200 that still carries the server's own headers is a broken symlink, not a deploy that never landed. +- **`--link-dest` on a remote deploy needs no bookkeeping: point it at `current`.** The flip happens after the upload, so at upload time `current` is still the previous release. A missing `--link-dest` is a warning and exit 0, not an error, so the first deploy degrades to a full copy on its own. Verified: same inode across releases through the symlink, link count 2. +- **The docker bridge gateway is inside the bridge subnet, so trusting the subnet trusts the host.** Verified here, not inherited: with `TRUSTED_PROXIES=172.18.0.0/16`, a request from this host straight to the container's bridge address forged `client_ip` successfully. Publishing no host port does not close it, because the bridge address is reachable from the host regardless. The value is now the subnet with the gateway's `/32` excluded, which is 16 CIDR blocks and cannot be written shorter. Re-tested both ways after: forgery ignored, real clients through Traefik still resolved. +- **`trusted_proxies` is a security boundary, and its RFC1918 default is only safe behind a proxy.** Trusting a range means believing `X-Forwarded-For` from anything in it, so on a host where the port is reachable directly, the default makes every device on the network able to forge the logged client address. `TRUSTED_PROXIES=` set-but-empty skips the default and trusts nothing, which is correct there; unset applies the default; an explicit list is exactly itself. All three verified. Binding to `127.0.0.1` does not close it, `docker-proxy` SNATs host traffic to the bridge gateway, which is inside RFC1918. - **A hard link keeps its inode's mode and ownership**, so `--chmod` and `--no-g` govern only newly transferred files. A badly moded file rides the link chain into every later release. `NO_LINK_DEST=1` mints fresh inodes. - **`DEPLOY_ROOT=... deploy/make-release.sh` does not select an environment.** The script sources its environment file with `set -a`, which exports every assignment in it and overwrites whatever the caller exported first, so the variable is set and then silently replaced. `ENV_FILE` selects the file, and the first argument overrides the root, because it is read afterwards. With two sites on one host the failure is not an error: it publishes to the other site. A named `ENV_FILE` that does not exist is a hard failure for the same reason. - **`content/` is an imported archive.** Prose, spelling, and style sweeps do not reach it, and `cspell.json` ignores it deliberately. diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index 9bf28c0..a0d8912 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -28,11 +28,9 @@ FAILED="$(mktemp)" CURLRC="" trap 'rm -f "$FAILED" ${CURLRC:+"$CURLRC"}' EXIT -# Staging sits behind Pangolin's auth gate, which a resource access token opens. -# The token pair goes into a curl config file rather than onto the command line, for two reasons. -# The check functions run under `export -f` and `xargs bash -c`, and bash cannot export an array, -# so a pair of -H arguments has no way to reach them intact. A command line is also world-readable -# in ps output for as long as the process lives, and this runs 1,245 of them. +# A resource access token opens the proxy's auth gate. +# It goes into a curl config file because bash cannot export an array to the parallel checks. +# A command line is also world-readable in ps output, and this runs 1,245 of them. if [ -n "${PANGOLIN_ACCESS_TOKEN_ID:-}" ] && [ -n "${PANGOLIN_ACCESS_TOKEN:-}" ]; then CURLRC="$(mktemp)" chmod 600 "$CURLRC" @@ -73,8 +71,8 @@ check_redirect() { esac # A redirect to a 404 is a broken redirect, so the destination is followed rather than trusted. dest=$(curl -s -o /dev/null -w '%{redirect_url}' --max-time 30 "${auth[@]}" "$BASE$url") - # Every destination in the contract is same-origin, and the credential is only ever sent to - # the origin it belongs to. A rule that one day redirects off-site must not mail the token there. + # The credential is only ever sent to the origin it belongs to. + # A rule that one day redirects off-site must not mail the token there. [ -n "$CURLRC" ] && [ "${dest#"$BASE"}" != "$dest" ] && dest_auth=(-K "$CURLRC") dcode=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${dest_auth[@]}" "$dest") # The media rule lands on an image, and a directory gains a trailing slash, so both answers are accepted. @@ -90,12 +88,22 @@ export BASE FAILED CURLRC echo "==> $BASE" # One request before the 1,245, because an auth gate turns a bad credential into a total failure. -# Without this the output is 1,245 lines saying the site is gone, when the site is fine and the -# token is wrong, and the two are indistinguishable from the far end of a CI log. -preflight=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${AUTH[@]}" "$BASE/") +# Otherwise the output reads as a vanished site rather than a wrong token. +preflight_headers=$(curl -s -o /dev/null -D- -w '%{http_code}' --max-time 30 "${AUTH[@]}" "$BASE/") +preflight="${preflight_headers##*$'\n'}" +header_of() { printf '%s' "$preflight_headers" | grep -i "^$1:" | tr -d '\r' | sed 's/^[^:]*: *//'; } + if [ "$preflight" != "200" ]; then echo "FAIL preflight: $BASE/ answered $preflight, expected 200" >&2 - if [ -n "$CURLRC" ]; then + # Own headers with no content behind them is the signature of a dangling current symlink. + # Caddy retains the last good config when the import disappears, so the headers stay correct. + # Reporting that as a release mismatch would send someone hunting a deploy that did land. + if [ -n "$(header_of x-blog-release)$(header_of x-blog-env)" ] && [ "$preflight" = "404" ]; then + echo " the server is up and holding a config, but serving no content, so 'current'" >&2 + echo " probably points at a release that is not on disk. Caddy keeps its last good" >&2 + echo " config when the import vanishes, which is why the headers below still look" >&2 + echo " right: env=$(header_of x-blog-env) release=$(header_of x-blog-release)" >&2 + elif [ -n "$CURLRC" ]; then echo " a token was sent, so check the pair is valid for this resource" >&2 else echo " no token was sent. If this site is behind the auth gate, set" >&2 @@ -104,6 +112,55 @@ if [ "$preflight" != "200" ]; then exit 1 fi +# Nothing in a response body says which environment answered. +# A proxy rule aimed at the wrong container returns a healthy 200 under the right hostname. +if [ -n "${EXPECT_SITE_ENV:-}" ]; then + got_env=$(printf '%s' "$preflight_headers" | grep -i '^x-blog-env:' | tr -d '\r' | sed 's/^[^:]*: *//') + if [ "$got_env" != "$EXPECT_SITE_ENV" ]; then + echo "FAIL preflight: $BASE/ is served by '${got_env:-}', expected '$EXPECT_SITE_ENV'" >&2 + echo " the hostname resolved to the wrong environment's container, or SITE_ENV is" >&2 + echo " unset on it. Checking the URL contract now would test the wrong site." >&2 + exit 1 + fi + echo "==> served by $got_env" +fi + +# Nothing else proves the rules answering are the ones just shipped, as no deploy restarts Caddy. +# A stale config serves the previous release's rules while the new content is already live. +read_release() { + curl -s -o /dev/null -D- --max-time 30 "${AUTH[@]}" "$BASE/" | + grep -i '^x-blog-release:' | tr -d '\r' | sed 's/^[^:]*: *//' +} + +got_release=$(printf '%s' "$preflight_headers" | grep -i '^x-blog-release:' | tr -d '\r' | sed 's/^[^:]*: *//') +if [ -n "${EXPECT_RELEASE:-}" ]; then + # The reload is asynchronous, so a check run straight after a deploy races it. + # Content is live instantly, while rules change on the next poll. + # The timeout still catches a container that is not watching, which never converges. + waited=0 + while [ "$got_release" != "$EXPECT_RELEASE" ] && [ "$waited" -lt "${RELOAD_TIMEOUT:-30}" ]; do + sleep 1 + waited=$((waited + 1)) + got_release=$(read_release) + done + if [ "$got_release" != "$EXPECT_RELEASE" ]; then + echo "FAIL preflight: after ${waited}s the rules are from release '${got_release:-}', expected '$EXPECT_RELEASE'" >&2 + echo " The content symlink moved but the config never followed, so the redirects" >&2 + echo " below would be checked against a config that was never deployed, and would" >&2 + echo " pass. Two causes, and the second is the likelier one on a server that has" >&2 + echo " been working:" >&2 + echo " - the container is not running 'caddy run --watch' at all; or" >&2 + echo " - it is, and the watcher is dead. It stops watching permanently after one" >&2 + echo " failed config load, logs nothing further, and reports healthy throughout." >&2 + echo " Anything that broke 'current' even briefly, including a test, does this." >&2 + echo " Only a container restart re-arms it." >&2 + exit 1 + fi + echo "==> rules from release $got_release${waited:+ (after ${waited}s)}" +elif [ -n "$got_release" ]; then + echo "==> rules from release $got_release" +fi + n_render=$(grep -c . "$CHECKS/golden-urls.txt") echo "==> checking $n_render URLs that must render" grep . "$CHECKS/golden-urls.txt" | xargs -P "$PARALLEL" -I{} bash -c 'check_render "$@"' _ {} diff --git a/deploy/Caddyfile b/deploy/Caddyfile index 17f252d..440d423 100644 --- a/deploy/Caddyfile +++ b/deploy/Caddyfile @@ -4,11 +4,35 @@ { admin off auto_https off + + # Trusting a range means believing X-Forwarded-For from anything inside it. + # The value is per-host, because docker subnets differ. + # An empty value trusts nothing, which is correct where Caddy is reachable without a proxy. + # The default keeps a host that omits the variable working. + servers { + trusted_proxies static {$TRUSTED_PROXIES:172.16.0.0/12 10.0.0.0/8 192.168.0.0/16} + } } :8080 { root * /srv/blog/current/site + # Only the container distinguishes environments, which all serve this bundle on one port. + # A proxy rule aimed at the wrong one answers healthily under the right hostname. + # The default matters, since an unset variable yields an empty header and still validates. + header X-Blog-Env "{$SITE_ENV:unset}" + + # Keeps a non-public copy of the site out of search indexes. + # The default is the value that is harmless on production. + # An unset staging container is still behind its auth gate. + # Production inheriting noindex would deindex the site silently. + header X-Robots-Tag "{$SITE_ROBOTS:index, follow}" + + # Names the release whose rules are answering, substituted at install time. + # Content follows the current symlink per request, while rules wait on a config reload. + # Without this a stale config passes the URL contract against rules never shipped. + header X-Blog-Release "@@RELEASE@@" + # One map serves three classes because their key spaces are disjoint. # Merging them keeps a single lookup on the hot path. map {path} {mapped} { diff --git a/deploy/README.md b/deploy/README.md index 53b1e85..62728a1 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -25,18 +25,23 @@ How the site is built, released, and served. The release is a **self-contained b This repo produces a release tree and the config that serves it. It never names a host path or an orchestrator variable, because the consumer may name both sides and the producer must name -neither. Five facts are the whole contract: +neither. Seven facts are the whole contract: 1. The deploy root is bind-mounted **read-only** at `/srv/blog`. Mount the **parent**, never `current`, because a symlink is resolved once at container creation and mounting it pins the container to whichever release was live then. 2. The site config is at `/srv/blog/current/Caddyfile`. -3. A stable per-container config directory is mounted at `/config`, holding a bootstrap that - imports (2). Mount `/data` as well to persist Caddy state across a recreate. The image ships - `/data/caddy`, so this is persistence rather than a startup requirement. +3. A stable per-container config directory is mounted **read-only** at `/config`, holding a + bootstrap that imports (2). Mount `/data` writable as well, and set `XDG_CONFIG_HOME=/data`. 4. Caddy binds `:8080`, plain HTTP, with `admin off` and `auto_https off`. TLS and the public listener belong to the fronting proxy. -5. The release tree is world-readable and world-traversable, so any uid can serve it. +5. Caddy runs with **`--watch`**. This is not optional. See + [Reloading without a restart](#reloading-without-a-restart). +6. The release tree is world-readable and world-traversable, so any uid can serve it. +7. The container sets `SITE_ENV` and `SITE_ROBOTS`, which the bundle stamps on every response + as `X-Blog-Env` and `X-Robots-Tag`, and `TRUSTED_PROXIES`. See + [Identifying the environment](#identifying-the-environment) and + [Trusting the proxy](#trusting-the-proxy). ## Building a release @@ -86,7 +91,8 @@ every build for that reason. | `REQUIRE_BROTLI=1` | Fails rather than shipping gzip-only. CI sets this. | | `NO_LINK_DEST=1` | Full copy instead of hard-linking from the previous release. | -`checks/check-live-urls.sh` reads two more, and neither reaches `make-release.sh`: +`checks/check-live-urls.sh` reads five more, and none of them reaches `make-release.sh`. Two +open the auth gate: | Variable | Effect | | --- | --- | @@ -99,6 +105,128 @@ mode-`600` config file rather than as `-H` arguments, which keeps the credential parallel checks run under. The token is sent to the base URL's own origin and to nothing else, so a redirect that one day points off-site cannot carry it away. +Three more assert that the thing answering is the thing that was just deployed, all checked in +the preflight before a single URL is requested: + +| Variable | Effect | +| --- | --- | +| `EXPECT_SITE_ENV` | Asserts the environment that answered, read from `X-Blog-Env`. | +| `EXPECT_RELEASE` | Asserts the release whose rules answered, read from `X-Blog-Release`. | +| `RELOAD_TIMEOUT` | Seconds to wait for that release to become live. Default 30. | + +## Reloading without a restart + +The container runs `caddy run --watch`, so a release goes live with **no restart**. The watcher +re-adapts the config on a timer and reloads it in process, and re-adapting re-executes every +`import`, which is how a new release's `Caddyfile` and `maps/*.map` are picked up through a +`/config/Caddyfile` that never itself changes. + +**`--watch` does not need the admin API.** `admin off` makes `caddy reload` impossible, which +looks like it should rule out reloading entirely, and does not: `caddy reload` POSTs to the admin +endpoint while the watcher reloads in process. The log prints `admin endpoint disabled` and +`watching config file for changes` together. + +**`XDG_CONFIG_HOME=/data` is required alongside it.** The watcher autosaves the adapted config on +every reload, to `$XDG_CONFIG_HOME/caddy`, which the image defaults to `/config`. A bind mount +over `/config` shadows the world-writable directory the image pre-creates there, so the autosave +has to `mkdir` and the outcome depends on the mount: + +| `/config` mount | Result | +| --- | --- | +| read-only, no `XDG_CONFIG_HOME` | an ERROR line **per reload**, one per deploy, reload still succeeds | +| read-only, `XDG_CONFIG_HOME=/data` | clean | +| writable, no `XDG_CONFIG_HOME` | silent, and Caddy writes `caddy/autosave.json` into the config directory | + +Mount `/config` read-only *and* set `XDG_CONFIG_HOME=/data`. Doing one without the other trades a +silent stray file for a per-deploy error, in the place a real error most needs to stand out. + +**Without `--watch` the failure is silent.** Caddy expands `import` at config-parse time and does +not watch the imported files, so the content symlink moves while the rules stay as they were when +Caddy last loaded. The URL check then passes against a config that was never deployed. That is why +the release stamps its own version into the config it ships with, as `X-Blog-Release`, and why +`check-live-urls.sh` compares it to `EXPECT_RELEASE` before checking a single URL. + +It **waits** for the match rather than sampling once. The reload is asynchronous, so a check +starting straight after a deploy races it and reads the previous release's config. `RELOAD_TIMEOUT` +bounds the wait, and a container that is not watching never converges, which is what turns a silent +staleness into a named failure. + +**Content and rules do not switch together.** `file_server` resolves `current` per request, so new +content is live instantly while the rules follow on the next poll, about a quarter of a second +later. For that window the new content is served under the previous release's rules. Harmless while +every rule is a redirect, since a stale redirect lands on a page that exists in both releases. It +stops being harmless if a rule ever *gates* content rather than redirecting it, and at that point +the flip has to become a restart again. + +## Identifying the environment + +Every environment runs the **same bundle on the same port** in its own container, so nothing in a +response says which one answered. A proxy rule aimed at the wrong container connects happily and +serves the wrong environment under the right hostname, returning a healthy `200`. That is a +failure a reader reports before a monitor notices. + +The bundle stamps two headers for that, taking both values from the container so the artifact +stays the same everywhere and still rolls back as one unit: + +| Container variable | Header | Values | +| --- | --- | --- | +| `SITE_ENV` | `X-Blog-Env` | `production`, `staging`, and the local mirrors | +| `SITE_ROBOTS` | `X-Robots-Tag` | `index, follow` or `noindex, nofollow` | + +Both are emitted at site level, outside the `route` block, which is what puts them on the error +path as well. Verified on all three response classes: `200` from `file_server`, `301` from a +`redir`, and `404` through `handle_errors`. + +**Both carry a default, because an unset `{$VAR}` is silent.** It expands to an empty header +rather than an error, and `caddy validate` still reports a valid configuration, so a missing +value would otherwise reach production unnoticed. `SITE_ENV` defaults to `unset`, which +`EXPECT_SITE_ENV` then fails on. + +**`SITE_ROBOTS` defaults to `index, follow`, which is deliberate and is not the safer-looking +choice.** The two failure directions are not symmetric: + +- A **staging** container missing the value is still behind its auth gate, so nothing reaches it + to index. The header is the second line of defence there, not the first. +- A **production** container that picked up `noindex` would deindex the site silently, and this + site's entire migration exists to preserve sixteen years of search ranking. Recovery is + measured in weeks of recrawling. + +So the default is the value that is harmless on production, and `noindex` is reachable only by +asking for it explicitly. + +`checks/check-live-urls.sh` asserts this when `EXPECT_SITE_ENV` is set, before it checks the +1,245 URLs, since checking the contract against the wrong environment proves nothing. + +## Trusting the proxy + +A proxy fronts Caddy in every environment, so the peer address is always the proxy and the real +client arrives in `X-Forwarded-For`. `trusted_proxies` is what makes Caddy believe it. Without it, +`client_ip` and `remote_ip` are both the proxy and the forwarded header has no effect at all. + +**The CIDRs come from the container, not the bundle.** The same artifact runs on hosts whose docker +subnets differ, so any literal in the bundle is wrong on one of them. + +**Trusting a range means believing `X-Forwarded-For` from anything inside it**, so the range is a +security boundary, and it should be no wider than what can actually reach the port. Three +behaviours, all verified: + +| `TRUSTED_PROXIES` | Result | +| --- | --- | +| unset | the bundle's default applies, all of RFC1918 | +| set but empty | the default is skipped and nothing is trusted | +| an explicit list | exactly those ranges | + +The default exists so a host that forgets the variable keeps working. It is not a safe value +everywhere: **it is only correct where a proxy is the only thing that can reach Caddy.** Where the +port is reachable directly, RFC1918 makes every device on the network a trusted proxy, and anything +there can forge the client address in the access log. `TRUSTED_PROXIES=` (present, blank) is the +right answer there, because a client that is not a proxy has no forwarded header worth honouring. + +Binding the port to `127.0.0.1` does **not** make direct access impossible. `docker-proxy` SNATs +host-originated traffic to the bridge gateway, which is itself inside RFC1918 and therefore inside +the default. Narrowing to the container subnet does not fix it either, since the gateway sits inside +that too and has to be excluded deliberately. + ## Layout ```text diff --git a/deploy/env.example b/deploy/env.example index ddab51d..a6f5357 100644 --- a/deploy/env.example +++ b/deploy/env.example @@ -41,7 +41,12 @@ CADDY_APPDATA=/path/to/container/appdata # The container to restart when config changes. # Caddy expands the site config and the map files at parse time and does not watch them, so a # release or a rollback that changes either is not live until the container restarts. -CADDY_CONTAINER=blog +# +# Environments are named `production` and `staging` everywhere, spelled out, with no `prod` or +# `stage` anywhere. The name is a value that gets compared, by EXPECT_SITE_ENV below and by the +# deploy, so a spelling that differs by environment fails a deploy for a reason that reads like +# an outage. The same two words name the container, the deploy root, and the header value. +CADDY_CONTAINER=blog-production # A Pangolin resource access token, read by check-live-urls.sh, set only for an environment # behind the auth gate. Staging keeps its gate on: it serves a byte-identical copy of the site, @@ -51,6 +56,12 @@ CADDY_CONTAINER=blog PANGOLIN_ACCESS_TOKEN_ID= PANGOLIN_ACCESS_TOKEN= +# The environment that must answer, compared against the X-Blog-Env header the bundle stamps. +# Every environment serves the same bundle on the same port, so a proxy rule aimed at the wrong +# container returns a healthy 200 from the wrong site under the right hostname. Checking the URL +# contract against that proves nothing, so the check refuses to start. +EXPECT_SITE_ENV=production + # Reserved for the remote deploy, which does not exist yet, and read only by CI when it does. # They are named here so the local file and the GitHub Environment describe the same shape. # A local run deploys to a path and needs none of them. diff --git a/deploy/make-release.sh b/deploy/make-release.sh index 0a017ec..dbdd98a 100755 --- a/deploy/make-release.sh +++ b/deploy/make-release.sh @@ -18,14 +18,8 @@ usage() { REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" # The deploy root and the base URL are the only host-specific values, and they pair per environment. -# CI passes both explicitly, and a local run reads them from an untracked file under secrets/. -# That directory is gitignored whole, so nothing naming this machine reaches a public repo. -# -# ENV_FILE selects which environment a local run targets, and sourcing is deliberately not -# conditional on the variable being unset: `set -a` exports every assignment in the file, so a -# DEPLOY_ROOT exported by the caller is overwritten rather than respected. Pointing ENV_FILE at -# the environment's own file is therefore the only way to switch environments, and passing the -# root as the first argument is the only way to override one, because that is read after this. +# ENV_FILE selects the environment, because `set -a` overwrites a value the caller exported. +# The first argument overrides the root, being read after this. DEFAULT_ENV_FILE="$REPO/secrets/.env" ENV_FILE="${ENV_FILE:-$DEFAULT_ENV_FILE}" if [ -f "$ENV_FILE" ]; then @@ -36,13 +30,23 @@ if [ -f "$ENV_FILE" ]; then set +a elif [ "$ENV_FILE" != "$DEFAULT_ENV_FILE" ]; then # The default file is optional, because CI passes every value explicitly and reads no file. - # A file named on the command line is not: a typo there would otherwise fall through to - # whatever the ambient environment holds, which on this host is the other site's root. + # A named file is not, since a typo would fall through to the other environment's root. echo "environment file not found: $ENV_FILE" >&2 exit 1 fi -ROOT="${1:-${DEPLOY_ROOT:-}}" +# This script installs to a local path, so a remote environment's DEPLOY_ROOT would be built here. +# The guard is on the fallback rather than the variable. +# An explicit first argument names a local path and is always honoured, which is what CI passes. +ROOT_ARG="${1:-}" +if [ -z "$ROOT_ARG" ] && [ -n "${DEPLOY_SSH_HOST:-}" ]; then + echo "$ENV_FILE names DEPLOY_SSH_HOST=$DEPLOY_SSH_HOST, so its DEPLOY_ROOT is a path on that" >&2 + echo "host and this script would create it here instead. Pass a local path as the first" >&2 + echo "argument to assemble a bundle for shipping, or use a local environment file." >&2 + exit 1 +fi + +ROOT="${ROOT_ARG:-${DEPLOY_ROOT:-}}" [ -n "$ROOT" ] || usage # CI passes the version so a release directory traces back to a commit rather than to a clock. @@ -136,6 +140,14 @@ rsync -a --no-g --chmod=D2755,F644 --delete "${LINK_SITE[@]}" public/ "$STAGE/si rsync -a --no-g --chmod=D2755,F644 --delete "${LINK_MAPS[@]}" "$REPO/deploy/maps/" "$STAGE/maps/" install -m 644 "$REPO/deploy/Caddyfile" "$STAGE/Caddyfile" +# Stamp the release into the config it ships with, so a response names the rules answering. +# A stale config otherwise passes the URL contract against rules that were never shipped. +sed -i "s/@@RELEASE@@/$VERSION/" "$STAGE/Caddyfile" +if grep -q "@@RELEASE@@" "$STAGE/Caddyfile"; then + echo "release stamp was not substituted into the shipped Caddyfile" >&2 + exit 1 +fi + # --chmod and --no-g govern only the files rsync newly transfers. # A file supplied by --link-dest keeps its original inode's mode, so the result is inspected rather than assumed. bad_files=$(find "$STAGE/site" "$STAGE/maps" -type f ! -perm -o=r | head -20) diff --git a/spec/secrets.json b/spec/secrets.json index 8a6d3b1..52ad5b6 100644 --- a/spec/secrets.json +++ b/spec/secrets.json @@ -18,13 +18,15 @@ "note": "The App-token secrets power the App-signed merge-bot (auto-merge that re-triggers downstream workflows), which every fleet repo runs. Used via actions/create-github-app-token with the client-id input (not the deprecated app-id). The CODEGEN_* name is historical, not codegen-specific." }, "environments": { - "note": "Not part of the baseline audit. Each deployment environment carries its own key so a staging deploy cannot reach production.", + "note": "Not part of the baseline audit. One key covers both environments, a deliberate decision recorded in OPERATIONS.md: the per-environment split only pays where the two keys never share a machine, and both sit on one workstation and in one secret store. The environment split still carries the deploy root, the base URL, and the staging-only access token, so it is not decorative.", "names": [ "staging", "production" ], "secrets": [ - "DEPLOY_SSH_PRIVATE_KEY" + "DEPLOY_SSH_PRIVATE_KEY", + "PANGOLIN_ACCESS_TOKEN_ID", + "PANGOLIN_ACCESS_TOKEN" ], "variables": [ "DEPLOY_SSH_HOST", @@ -32,6 +34,7 @@ "DEPLOY_SSH_KNOWN_HOSTS", "DEPLOY_ROOT", "HUGO_BASEURL" - ] + ], + "secretsNote": "PANGOLIN_ACCESS_TOKEN_ID and PANGOLIN_ACCESS_TOKEN are set on staging only. Staging keeps its auth gate on, and production must answer unauthenticated, so both are absent there and checks/check-live-urls.sh then sends no credential." } } From b5d37442824c33019666ba15120f0bd53ca5e607 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 12:57:16 -0700 Subject: [PATCH 03/11] Anchor ENV_FILE under the repo and correct the restart note A relative ENV_FILE resolved against the caller's working directory while the default resolved against the repo, so the same name meant different files depending on where the script ran. The container comment described a restart as the way a config change goes live, which the in-process reload replaced. Co-Authored-By: Claude Opus 5 (1M context) --- deploy/env.example | 57 ++++++++++++++++++------------------------ deploy/make-release.sh | 5 ++++ 2 files changed, 29 insertions(+), 33 deletions(-) diff --git a/deploy/env.example b/deploy/env.example index a6f5357..5f81f59 100644 --- a/deploy/env.example +++ b/deploy/env.example @@ -1,17 +1,15 @@ -# Copy to secrets/.env and set for this host. Every value here names a particular -# machine rather than the project, which is why secrets/ is gitignored as a whole directory and -# this template lives outside it. CI sets the deploy values from environment secrets and never -# reads a file. +# Copy to secrets/.env and set for this host. +# Every value here names a machine rather than the project, so secrets/ is gitignored whole. +# CI sets the deploy values from environment secrets and reads no file. # # One file per environment, selected by ENV_FILE: # secrets/.env the default, read when ENV_FILE is unset # secrets/staging.env ENV_FILE=secrets/staging.env deploy/make-release.sh # -# The file is sourced with `set -a`, so every assignment in it is exported and a variable the -# caller exported first is overwritten. Selecting the file is therefore how an environment is -# chosen, and the first argument to make-release.sh is how its root is overridden. A named file -# that does not exist is a hard failure rather than a fall-through to the ambient environment, -# because on a host running two sites the ambient value is the other site's root. +# The file is sourced with `set -a`, which overwrites a variable the caller exported first. +# Selecting the file is therefore how an environment is chosen. +# The first argument to make-release.sh is how its root is overridden. +# A named file that does not exist is a hard failure rather than a fall-through. # # Naming convention: the prefix names whatever owns the value, not whatever reads it. # HUGO_ is fixed by Hugo, which maps HUGO_ onto its own config natively. @@ -20,50 +18,43 @@ # PANGOLIN_ is the proxy, which owns the credential that opens its auth gate. # Written by every release, and mounted read-only by the container at /srv/blog. -# make-release.sh also takes it as a first argument, which wins over this value. -# CI sets the same name per environment, where the deploy is an rsync rather than a local copy. +# The first argument to make-release.sh wins over this value. DEPLOY_ROOT=/path/to/deploy/root # Must be set for anything that is not production. -# The base URL is baked into the canonical tag, the feed links, and every absolute permalink, -# so a mirror built without it serves pages that all point back at production while every gate -# still passes. +# The base URL is baked into the canonical tag, the feed links, and every absolute permalink. +# A mirror built without it serves pages pointing back at production, and every gate still passes. HUGO_BASEURL=https://blog.example.com/ # The container's persistent state root, deliberately outside DEPLOY_ROOT. # Two directories hang off it, and a release writes neither: # /config mounted at /config, holding the bootstrap Caddyfile -# /data mounted at /data, persisting Caddy state across a recreate -# No script reads this. It is recorded so a rebuild does not depend on someone remembering -# where the bootstrap goes, since the release bundle deliberately does not carry it. +# /data mounted at /data, holding Caddy state and the reload autosave +# No script reads this, so it is recorded to keep a rebuild from depending on memory. CADDY_APPDATA=/path/to/container/appdata -# The container to restart when config changes. -# Caddy expands the site config and the map files at parse time and does not watch them, so a -# release or a rollback that changes either is not live until the container restarts. +# The container serving this environment. +# A release needs no restart, because Caddy reloads its config in process. +# Restarting is the remedy when the watcher dies, which it does silently after one failed load. # -# Environments are named `production` and `staging` everywhere, spelled out, with no `prod` or -# `stage` anywhere. The name is a value that gets compared, by EXPECT_SITE_ENV below and by the -# deploy, so a spelling that differs by environment fails a deploy for a reason that reads like -# an outage. The same two words name the container, the deploy root, and the header value. +# Environments are named production and staging, spelled out, with no prod or stage anywhere. +# The name is compared, by EXPECT_SITE_ENV below and by the deploy. +# A spelling that differs by environment fails a deploy for a reason that reads like an outage. CADDY_CONTAINER=blog-production -# A Pangolin resource access token, read by check-live-urls.sh, set only for an environment -# behind the auth gate. Staging keeps its gate on: it serves a byte-identical copy of the site, -# and a public one is a duplicate handed to every crawler, which is the one thing a migration -# about preserving URLs should not do. The check punches through with these instead. +# A resource access token, read by check-live-urls.sh, for an environment behind the auth gate. +# Staging keeps its gate on, because it serves a byte-identical copy of the public site. # Set both or neither, and leave both unset for a site that is public. PANGOLIN_ACCESS_TOKEN_ID= PANGOLIN_ACCESS_TOKEN= # The environment that must answer, compared against the X-Blog-Env header the bundle stamps. -# Every environment serves the same bundle on the same port, so a proxy rule aimed at the wrong -# container returns a healthy 200 from the wrong site under the right hostname. Checking the URL -# contract against that proves nothing, so the check refuses to start. +# A proxy rule aimed at the wrong container returns a healthy 200 under the right hostname. +# Checking the URL contract against that proves nothing, so the check refuses to start. EXPECT_SITE_ENV=production -# Reserved for the remote deploy, which does not exist yet, and read only by CI when it does. -# They are named here so the local file and the GitHub Environment describe the same shape. +# Read by the deploy workflow, which resolves them from the GitHub Environment rather than a file. +# They are named here so the local file and the environment describe the same shape. # A local run deploys to a path and needs none of them. #DEPLOY_SSH_HOST= #DEPLOY_SSH_USER= diff --git a/deploy/make-release.sh b/deploy/make-release.sh index dbdd98a..a274277 100755 --- a/deploy/make-release.sh +++ b/deploy/make-release.sh @@ -22,6 +22,11 @@ REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" # The first argument overrides the root, being read after this. DEFAULT_ENV_FILE="$REPO/secrets/.env" ENV_FILE="${ENV_FILE:-$DEFAULT_ENV_FILE}" +# Anchored under the repo, so a relative name resolves the same from any working directory. +case "$ENV_FILE" in +/*) ;; +*) ENV_FILE="$REPO/$ENV_FILE" ;; +esac if [ -f "$ENV_FILE" ]; then echo "==> environment: $ENV_FILE" set -a From aff8581c1641896bce5fb1ca5b53b810f5aeef45 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 13:33:30 -0700 Subject: [PATCH 04/11] Scope the staging-only secrets and correct the environment prose The manifest listed the access token beside a credential every environment carries, which reads as required in both. A name audit would then report it missing from production, where it is deliberately absent. The README described configuration as coming from one file, which the per-environment selection replaced. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 2 +- TODO.md | 1 + spec/secrets.json | 13 +++++++++---- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 883a44b..b25ad3b 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ deploy/make-release.sh checks/check-live-urls.sh "$HUGO_BASEURL" ``` -The deploy root and the base URL come from an untracked `secrets/.env`, copied from [deploy/env.example][env-example]. The whole `secrets/` directory is gitignored, so host-specific values stay out of the published history. +The deploy root and the base URL come from an untracked file per environment under `secrets/`, copied from [deploy/env.example][env-example] and selected with `ENV_FILE`. The whole `secrets/` directory is gitignored, so host-specific values stay out of the published history. ## 3rd Party Tools diff --git a/TODO.md b/TODO.md index faf9cdc..cad20f9 100644 --- a/TODO.md +++ b/TODO.md @@ -84,6 +84,7 @@ Each of these was hit or nearly hit, and each is cheap to re-trip. - **Do not name any workflow `build-*-task.yml`** while the repo declares `source-only`, since `detect` is literally `["no build-*-task.yml"]`. - **Do not edit `.markdownlint-cli2.jsonc`, `repo-config/configure.sh`, or the two ruleset payloads.** They are carried verbatim and byte-matched against the hub. Scope a glob in the workflow instead. A reviewer finding a real defect in one of them is answered by declining locally and filing it at the hub, never by editing the file to satisfy the review. - **The hub's `main` can promote while a convergence pull request is open**, so ground truth moves underneath work that was correct when it started. It happened twice in one session on 2026-08-03, and the second time added drift the branch could not have known about. Re-run the audit against the hub ref actually carried before claiming convergence, and name that ref in the change, or the claim ages into a false one. +- **A Copilot review hides findings in the review body, where the thread API cannot see them.** The `reviewThreads` query returns line threads only, so a review carrying `Suppressed comments (N)` in a `
` block reports zero unresolved while real findings sit unread. Read the review body itself, not just the threads, before calling a review loop finished. - **`gh pr merge --delete-branch` on a `develop -> main` promotion deletes `develop`.** Use a plain `gh pr merge --merge`. ## Reference diff --git a/spec/secrets.json b/spec/secrets.json index 52ad5b6..af7078c 100644 --- a/spec/secrets.json +++ b/spec/secrets.json @@ -24,9 +24,7 @@ "production" ], "secrets": [ - "DEPLOY_SSH_PRIVATE_KEY", - "PANGOLIN_ACCESS_TOKEN_ID", - "PANGOLIN_ACCESS_TOKEN" + "DEPLOY_SSH_PRIVATE_KEY" ], "variables": [ "DEPLOY_SSH_HOST", @@ -35,6 +33,13 @@ "DEPLOY_ROOT", "HUGO_BASEURL" ], - "secretsNote": "PANGOLIN_ACCESS_TOKEN_ID and PANGOLIN_ACCESS_TOKEN are set on staging only. Staging keeps its auth gate on, and production must answer unauthenticated, so both are absent there and checks/check-live-urls.sh then sends no credential." + "secretsNote": "The 'secrets' and 'variables' lists are required in every environment named above. 'environmentSecrets' names what one environment carries and another does not, so a name audit does not read a staging-only credential as missing from production. Staging keeps its auth gate on and production answers unauthenticated, so the access token exists on staging alone and checks/check-live-urls.sh sends no credential where the pair is absent.", + "environmentSecrets": { + "staging": [ + "PANGOLIN_ACCESS_TOKEN_ID", + "PANGOLIN_ACCESS_TOKEN" + ], + "production": [] + } } } From 4ba6bb35f1e723f33bd59e02fe244fc515a17c4b Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 13:42:36 -0700 Subject: [PATCH 05/11] Assert the release placeholder and constrain the environment name sed reports success when it matches nothing, so a Caddyfile that lost the placeholder shipped unstamped and the live check then blamed a dead config watcher for a bundle that never carried a release id. The environment name selects a GitHub Environment and lands in a remote path, and a workflow_call caller is not bound by the dispatch choice list. It is asserted in its own job, because the environment binding resolves before any step in the deploying job runs. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/deploy-site-task.yml | 21 +++++++++++++++++++++ deploy/make-release.sh | 7 +++++++ 2 files changed, 28 insertions(+) diff --git a/.github/workflows/deploy-site-task.yml b/.github/workflows/deploy-site-task.yml index 0ece398..8fd7f9e 100644 --- a/.github/workflows/deploy-site-task.yml +++ b/.github/workflows/deploy-site-task.yml @@ -16,10 +16,31 @@ env: jobs: + # The name selects a GitHub Environment and lands in a remote path, and a workflow_call caller + # is not bound by the dispatch choice list. + # A separate job, because the environment binding below resolves before any step runs. + assert-environment: + name: Assert environment name job + runs-on: ubuntu-latest + steps: + - name: Assert environment is known step + env: + ENVIRONMENT: ${{ inputs.environment }} + run: | + set -Eeuo pipefail + case "$ENVIRONMENT" in + production | staging) ;; + *) + echo "::error::environment must be production or staging; got '$ENVIRONMENT'." + exit 1 + ;; + esac + # Host-specific values come from the environment, so this file names no host, path, or address. deploy: name: Deploy site job runs-on: ubuntu-latest + needs: [ assert-environment ] environment: ${{ inputs.environment }} permissions: contents: read diff --git a/deploy/make-release.sh b/deploy/make-release.sh index a274277..8c09410 100755 --- a/deploy/make-release.sh +++ b/deploy/make-release.sh @@ -147,6 +147,13 @@ install -m 644 "$REPO/deploy/Caddyfile" "$STAGE/Caddyfile" # Stamp the release into the config it ships with, so a response names the rules answering. # A stale config otherwise passes the URL contract against rules that were never shipped. +# Asserted before substituting, because sed reports success when it matches nothing. +# A Caddyfile that lost the placeholder would otherwise ship unstamped, and the live check would +# then blame a dead config watcher for a bundle that never carried a release id. +if ! grep -q "@@RELEASE@@" "$REPO/deploy/Caddyfile"; then + echo "deploy/Caddyfile carries no @@RELEASE@@ placeholder to stamp" >&2 + exit 1 +fi sed -i "s/@@RELEASE@@/$VERSION/" "$STAGE/Caddyfile" if grep -q "@@RELEASE@@" "$STAGE/Caddyfile"; then echo "release stamp was not substituted into the shipped Caddyfile" >&2 From 8770acc574b5fa36f168ae67aaa587422b8309b9 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 13:43:13 -0700 Subject: [PATCH 06/11] Record the reviewer login trap that reads as a clean review Co-Authored-By: Claude Opus 5 (1M context) --- TODO.md | 1 + 1 file changed, 1 insertion(+) diff --git a/TODO.md b/TODO.md index cad20f9..995989b 100644 --- a/TODO.md +++ b/TODO.md @@ -84,6 +84,7 @@ Each of these was hit or nearly hit, and each is cheap to re-trip. - **Do not name any workflow `build-*-task.yml`** while the repo declares `source-only`, since `detect` is literally `["no build-*-task.yml"]`. - **Do not edit `.markdownlint-cli2.jsonc`, `repo-config/configure.sh`, or the two ruleset payloads.** They are carried verbatim and byte-matched against the hub. Scope a glob in the workflow instead. A reviewer finding a real defect in one of them is answered by declining locally and filing it at the hub, never by editing the file to satisfy the review. - **The hub's `main` can promote while a convergence pull request is open**, so ground truth moves underneath work that was correct when it started. It happened twice in one session on 2026-08-03, and the second time added drift the branch could not have known about. Re-run the audit against the hub ref actually carried before claiming convergence, and name that ref in the change, or the claim ages into a false one. +- **The Copilot reviewer's login differs by API, and a wrong-form filter reads as a clean review.** REST reports `copilot-pull-request-reviewer[bot]`, GraphQL omits the suffix. A filter written in the other form matches nothing, and an empty result is indistinguishable from no findings. Assert the filter matched before trusting what it returned. - **A Copilot review hides findings in the review body, where the thread API cannot see them.** The `reviewThreads` query returns line threads only, so a review carrying `Suppressed comments (N)` in a `
` block reports zero unresolved while real findings sit unread. Read the review body itself, not just the threads, before calling a review loop finished. - **`gh pr merge --delete-branch` on a `develop -> main` promotion deletes `develop`.** Use a plain `gh pr merge --merge`. From 0f03139b7b112808a60dd0f3a37d5ead372e36d2 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 13:50:49 -0700 Subject: [PATCH 07/11] Describe the accepted statuses, the env file, and the default name The redirect failure named one status while the check accepted two, and the usage text described a deploy root the environment file selection replaced. The setup instruction named only the per-environment form, so a single environment produced a file nothing reads. Co-Authored-By: Claude Opus 5 (1M context) --- checks/check-live-urls.sh | 2 +- deploy/README.md | 10 ++++++---- deploy/make-release.sh | 3 ++- 3 files changed, 9 insertions(+), 6 deletions(-) diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index a0d8912..0425fc7 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -65,7 +65,7 @@ check_redirect() { case "$code" in 301 | 308) ;; *) - echo "redirect $url expected 301, got $code" >>"$FAILED" + echo "redirect $url expected 301 or 308, got $code" >>"$FAILED" return ;; esac diff --git a/deploy/README.md b/deploy/README.md index 62728a1..87dbce8 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -51,10 +51,12 @@ checks/check-live-urls.sh "$HUGO_BASEURL" ``` The deploy root and the base URL are the only host-specific values, and they pair per -environment. Copy [`env.example`](./env.example) to `secrets/.env` and set both. -`secrets/` is gitignored as a whole directory, so a value naming one machine cannot reach a -public repo by being added to a file nobody remembered to ignore. CI passes them explicitly -instead, which keeps a pipeline run self-describing: +environment. Copy [`env.example`](./env.example) to `secrets/.env`, which is the file read when +`ENV_FILE` is unset, and add `secrets/.env` for each further environment. A single +environment therefore needs `secrets/.env` and nothing else, since a differently named file is +read only when `ENV_FILE` names it. `secrets/` is gitignored as a whole directory, so a value +naming one machine cannot reach a public repo by being added to a file nobody remembered to +ignore. CI passes them explicitly instead, which keeps a pipeline run self-describing: ```sh HUGO_BASEURL= deploy/make-release.sh "$(git rev-parse --short HEAD)" diff --git a/deploy/make-release.sh b/deploy/make-release.sh index 8c09410..4f07df1 100755 --- a/deploy/make-release.sh +++ b/deploy/make-release.sh @@ -11,7 +11,8 @@ KEEP_RELEASES=10 usage() { echo "usage: $0 [deploy-root] [version]" >&2 - echo " deploy-root defaults to DEPLOY_ROOT, from the environment or secrets/.env" >&2 + echo " deploy-root defaults to DEPLOY_ROOT, from the environment or \$ENV_FILE" >&2 + echo " ENV_FILE defaults to secrets/.env, and a relative path resolves against the repo" >&2 exit 2 } From 3cf9c349297f3d6ac06abb2944575830d963afd1 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 13:57:29 -0700 Subject: [PATCH 08/11] Require an origin boundary before sending the access token The base URL has its trailing slash stripped, so a prefix test also matched a host that merely starts with it. A redirect to a lookalike registered as an attacker's subdomain would have received the credential. Co-Authored-By: Claude Opus 5 (1M context) --- checks/check-live-urls.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index 0425fc7..9fedabb 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -73,7 +73,13 @@ check_redirect() { dest=$(curl -s -o /dev/null -w '%{redirect_url}' --max-time 30 "${auth[@]}" "$BASE$url") # The credential is only ever sent to the origin it belongs to. # A rule that one day redirects off-site must not mail the token there. - [ -n "$CURLRC" ] && [ "${dest#"$BASE"}" != "$dest" ] && dest_auth=(-K "$CURLRC") + # The match needs an origin boundary, since a bare prefix also accepts a host that merely + # starts with this one, such as a lookalike registered as an attacker's subdomain. + if [ -n "$CURLRC" ]; then + case "$dest" in + "$BASE" | "$BASE"/*) dest_auth=(-K "$CURLRC") ;; + esac + fi dcode=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "${dest_auth[@]}" "$dest") # The media rule lands on an image, and a directory gains a trailing slash, so both answers are accepted. case "$dcode" in From 421737ea66e19ebfd90ae75c0e2d96f5b05ba21b Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 14:07:05 -0700 Subject: [PATCH 09/11] Constrain the release version to a path-safe and substitution-safe form The value becomes a directory name, a symlink target, and a sed replacement, so a separator, a traversal, or a substitution metacharacter would place the release outside releases/ or corrupt the release stamp. Co-Authored-By: Claude Opus 5 (1M context) --- deploy/make-release.sh | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/deploy/make-release.sh b/deploy/make-release.sh index 4f07df1..63f9a43 100755 --- a/deploy/make-release.sh +++ b/deploy/make-release.sh @@ -58,6 +58,15 @@ ROOT="${ROOT_ARG:-${DEPLOY_ROOT:-}}" # CI passes the version so a release directory traces back to a commit rather than to a clock. VERSION="${2:-$(date -u +%Y%m%d-%H%M%S)}" +# Constrained because the value becomes a directory name, a symlink target, and a sed replacement. +# A separator or a traversal would place the release outside releases/ or corrupt the stamp. +case "$VERSION" in +"" | *[!A-Za-z0-9._-]* | *..* | -*) + echo "version must be one or more of A-Z a-z 0-9 . _ -, without '..' or a leading '-'" >&2 + exit 1 + ;; +esac + command -v hugo >/dev/null || { echo "hugo not found on PATH" >&2 exit 1 From fd9ffb014b9647d117b9d23b7ac6199cbdb039ec Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 14:13:00 -0700 Subject: [PATCH 10/11] Record the deploy scripts' Linux dependency ceiling The scripts use GNU and bash 4.4 constructs that a stock macOS lacks, and the platform rule requires a narrowed platform to be recorded so it reads as a dependency ceiling rather than an oversight. Co-Authored-By: Claude Opus 5 (1M context) --- deploy/README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/deploy/README.md b/deploy/README.md index 87dbce8..4d7759f 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -15,6 +15,12 @@ How the site is built, released, and served. The release is a **self-contained b | `bash` 4.4+ | all scripts | arrays, `mapfile` | already present | | `docker` | serving | running Caddy | orchestrated elsewhere | +**These scripts run on Linux, which is a dependency ceiling rather than an omission.** They use +`mv -Tf`, `find -printf`, `mapfile`, and `sed -i` without an argument, all of which are GNU or +bash 4.4 constructs absent from a stock macOS. Every consumer is Linux already: CI builds there, +the containers serve there, and the deploy account receives there. Editing is unaffected, and a +macOS or Windows contributor runs them through a container or a remote Linux host. + **Hugo must be the `extended` build**, and `hugo version` reports `+extended` when it is. Debian's archive does not carry a useful version, so install from the upstream release or via snap. **`brotli` is easy to miss and degrades silently.** The Caddyfile serves `precompressed br gzip`, so without the binary every text response falls back to gzip while the site keeps working and nothing errors. `make-release.sh` warns loudly when it is absent, and `REQUIRE_BROTLI=1` turns that into a hard failure. **CI must set it.** Measured on the home page: 18,845 bytes raw, 6,395 gzip, 5,105 brotli. From aec533cb8f56158d5b45bb167def569356b05bcc Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Tue, 4 Aug 2026 14:19:20 -0700 Subject: [PATCH 11/11] Report an unreachable host as transport rather than as a status A name that does not resolve produced a status code, which the preflight then explained as a bad credential or a broken symlink. Co-Authored-By: Claude Opus 5 (1M context) --- checks/check-live-urls.sh | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/checks/check-live-urls.sh b/checks/check-live-urls.sh index 9fedabb..eba83c5 100755 --- a/checks/check-live-urls.sh +++ b/checks/check-live-urls.sh @@ -25,8 +25,9 @@ for list in golden-urls.txt redirect-urls.txt; do done FAILED="$(mktemp)" +CURLERR="$(mktemp)" CURLRC="" -trap 'rm -f "$FAILED" ${CURLRC:+"$CURLRC"}' EXIT +trap 'rm -f "$FAILED" "$CURLERR" ${CURLRC:+"$CURLRC"}' EXIT # A resource access token opens the proxy's auth gate. # It goes into a curl config file because bash cannot export an array to the parallel checks. @@ -95,7 +96,13 @@ echo "==> $BASE" # One request before the 1,245, because an auth gate turns a bad credential into a total failure. # Otherwise the output reads as a vanished site rather than a wrong token. -preflight_headers=$(curl -s -o /dev/null -D- -w '%{http_code}' --max-time 30 "${AUTH[@]}" "$BASE/") +# Transport failures are separated from HTTP ones, since a name that does not resolve otherwise +# reports as a status code and gets diagnosed as a credential or a symlink. +if ! preflight_headers=$(curl -sS -o /dev/null -D- -w '%{http_code}' --max-time 30 "${AUTH[@]}" "$BASE/" 2>"$CURLERR"); then + echo "FAIL preflight: $BASE/ could not be reached, so nothing below was checked" >&2 + sed 's/^/ /' "$CURLERR" >&2 + exit 1 +fi preflight="${preflight_headers##*$'\n'}" header_of() { printf '%s' "$preflight_headers" | grep -i "^$1:" | tr -d '\r' | sed 's/^[^:]*: *//'; }