diff --git a/.gitignore b/.gitignore index 6872d64..8c5d34f 100644 --- a/.gitignore +++ b/.gitignore @@ -31,9 +31,12 @@ __pycache__/ # Each names one particular machine rather than the project. # The whole directory is ignored so a value added later lands ignored by default. # `deploy/env.example` is the committed template and sits outside the directory. +# The last pattern is the backstop for one written outside the directory, matching the +# `..env` shape those files are named for rather than a single literal name. +# `deploy/env.example` does not end in `.env`, so it is unaffected. secrets/ **/secrets -.env +*.env # The working copies of the host channel described in OPERATIONS.md. # They carry server internals, and the host's own backup is what makes them durable. diff --git a/OPERATIONS.md b/OPERATIONS.md index 5ef1031..225a56e 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -8,16 +8,16 @@ Four environments, in two pairs. Each pair is one publish site and one staging s | Environment | Address | Fronted by | Purpose | | --- | --- | --- | --- | -| Local publish mirror | a private hostname, set in `secrets/.env` | Traefik, on the maintainer's own network | Proves the artifact. The redirect rules, the maps, and the release mechanics. | -| Local staging mirror | a second private hostname, set in `secrets/staging.env` | Traefik | Proves that two environments on one host stay independent, before that matters on a server. | -| Staging | `blog.vps.insanegenius.net`, behind the auth gate | Pangolin | Proves the infrastructure. Routing, TLS, and the deploy path. | -| Production | `blog.insanegenius.com` | Pangolin | The public site. | +| Local publish mirror | a private hostname, set in `secrets/local.production.env` | Traefik, on the maintainer's own network | Proves the artifact. The redirect rules, the maps, and the release mechanics. | +| Local staging mirror | a second private hostname, set in `secrets/local.staging.env` | Traefik | Proves that two environments on one host stay independent, before that matters on a server. | +| Staging | `blog.vps.insanegenius.net`, behind the auth gate, set in `secrets/vps.staging.env` | Pangolin | Proves the infrastructure. Routing, TLS, and the deploy path. | +| Production | `blog.insanegenius.com`, set in `secrets/vps.production.env` | Pangolin | The public site. | The local mirrors are not staging. They run the same bundle against the same web server, so they catch a broken redirect or a bad permission for free, but they exercise none of the routing, authentication, or certificate machinery that only exists on the VPS. Passing locally says the artifact is right. It says nothing about whether the server in front of it is. **The two words are `production` and `staging`, spelled out, in every position.** No `prod`, no `stage`. The same two name the container, the deploy root, the environment file, the `X-Blog-Env` value, and the GitHub Environment. This is not tidiness: the environment name is a value that gets **compared**, by `EXPECT_SITE_ENV` and by the deploy, so a spelling that differs in one position fails a deploy for a reason that reads like an outage. The local mirrors prefix the same words, `mirror-production` and `mirror-staging`, so a header names exactly one of the four environments in the fleet. -Each environment is one file under `secrets/`, selected with `ENV_FILE`, holding the deploy root, the base URL, and the container name. Selecting the file is how an environment is chosen: the file is sourced with `set -a`, so it overwrites a `DEPLOY_ROOT` the caller exported and setting that variable by hand does not switch anything. A named file that does not exist is a hard failure rather than a fall-through, because on a host serving two sites the ambient value is the other site's root. +Each environment is one file under `secrets/`, named `..env`, selected with `ENV_FILE`, and holding the deploy root, the base URL, and the container name. The name carries both halves because the two pairs differ in server as well as environment, so a file says which machine it describes rather than leaving that to the value inside it, and the four in the table above are the four files. `secrets/local.production.env` is the one read when `ENV_FILE` is unset. Selecting the file is how an environment is chosen: the file is sourced with `set -a`, so it overwrites a `DEPLOY_ROOT` the caller exported and setting that variable by hand does not switch anything. A named file that does not exist is a hard failure rather than a fall-through, because on a host serving two sites the ambient value is the other site's root. **The staging FQDN sits under the VPS wildcard deliberately.** `blog.vps.insanegenius.net` needs no new certificate and no new DNS record, and it keeps the staging name off the production domain. @@ -46,9 +46,32 @@ This site has served the same domain across earlier platforms, so its whole oper **The contract is ground truth.** [`checks/golden-urls.txt`](./checks/golden-urls.txt) and [`checks/redirect-urls.txt`](./checks/redirect-urls.txt) record URLs verified with a live request, not predicted from the content tree. The lists are **append-only**: nothing legitimately removes a URL the site has served, so a change that would drop one is a change to reject rather than a list to shorten. A list-driven check also carries a length floor, or a truncated list passes while checking almost nothing. +## The Migration Record + +**The migration is documented once, as a post on the site, and that post is the artifact to reference.** [`content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md`](./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md) holds how the URL surface was captured, why the contract splits into a render half and a redirect half, why the Blogger permalink map needs more entries than the posts it covers, why the media had to come from the export tar and be hash-verified, and which Hugo taxonomy default moves every archive to a new address without reporting anything. + +**Read it before changing anything under [`checks/`](./checks/) or [`deploy/maps/`](./deploy/maps/).** Both hold values that no code derives and no test explains, and the reasoning behind them is in the post rather than beside them. Cite the post rather than restating it. This file is the procedure and the post is the account of how the procedure came to be, so where the two disagree this file governs what to do while the post explains why the check exists. + +**The post is content, so it sits under the URL contract.** Editing it moves nothing. Renaming it or taking it down breaks an address the site serves. A fact in it that proves wrong is corrected in the post rather than footnoted here. + +### Rebuilding from the Exports + +Everything derived is in this repository. Everything it was derived *from* is in a capture directory outside it, which is where a rebuild starts. **The capture path is `CAPTURE_ROOT` in `secrets/local.production.env`**, recorded alongside the other values that name a machine rather than the project, so it is read from there rather than searched for. The capture is not a git repository, so it has no history to revert to, and it is read-only in normal use. + +| Under the capture | Holds | Recoverable | +| --- | --- | --- | +| `export/raw/` | the WordPress content export, WXR XML | yes, from the WordPress account while it exists | +| `export/media-tar/` | the media export, the only trustworthy copy of the images | yes, from the same place | +| `mirror/` | a crawl of the old platform as it served, including the media it linked from other hosts | no, once the old hosting ends | +| `inventory/` | the URL and media inventories derived from that crawl | no, for the same reason | + +The two exports are the only inputs a person has to fetch, and `EXPORT-INSTRUCTIONS.md` at the root of the capture records which two menu items produce them and the counts each has to reconcile against. The counts are the point, because a partial export is the common way a migration loses posts without reporting anything. + +[`checks/build-redirects.py`](./checks/build-redirects.py) takes the capture directory as its one argument and rebuilds everything under `deploy/maps/` from it. It selects the export **by content** rather than by filename and fails unless exactly one candidate holds published posts, because the capture also holds a media-only export whose zero posts produce empty maps that are indistinguishable from working ones until the redirects are live. + ## Local Verification Before a Pull Request -**CI cannot prove a redirect.** The validation workflow builds the site and checks the render half of the contract, which is every URL that must return a page. The other 917 URLs are the web server's job, and nothing in a build exercises them. A change to the Caddy config or to a generated map is therefore invisible to CI: the workflow goes green while the redirect it broke stays broken until someone follows a sixteen-year-old link. +**CI cannot prove a redirect.** The validation workflow builds the site and checks the render half of the contract, which is every URL that must return a page. Most of the contract is not pages, and those URLs are the web server's job, which nothing in a build exercises. A change to the Caddy config or to a generated map is therefore invisible to CI: the workflow goes green while the redirect it broke stays broken until someone follows a sixteen-year-old link. So release to the local mirror and run the live check **before** opening a pull request that touches any of these: @@ -60,7 +83,7 @@ So release to the local mirror and run the live check **before** opening a pull | `hugo.yaml`, `layouts/` | Permalink and taxonomy changes move URLs underneath the redirects that point at them. | ```sh -set -a; . secrets/.env; set +a +set -a; . secrets/local.production.env; set +a deploy/make-release.sh checks/check-live-urls.sh "$HUGO_BASEURL" ``` @@ -68,8 +91,8 @@ checks/check-live-urls.sh "$HUGO_BASEURL" Against the staging mirror, name its file in both places, since the sourced values and the ones `make-release.sh` reads must describe the same environment: ```sh -set -a; . secrets/staging.env; set +a -ENV_FILE=secrets/staging.env deploy/make-release.sh +set -a; . secrets/local.staging.env; set +a +ENV_FILE=secrets/local.staging.env deploy/make-release.sh checks/check-live-urls.sh "$HUGO_BASEURL" ``` @@ -91,11 +114,11 @@ EXPECT_RELEASE= checks/check-live-urls.sh "$HUGO_BASEURL" Sourcing the environment file first puts the deploy root and the base URL in the environment, so no literal value is typed. `make-release.sh` then needs no arguments, because its deploy root falls back to `$DEPLOY_ROOT` and its version falls back to a timestamp. It still accepts both, and [Deploying](#deploying) below passes them explicitly, which is what CI does so a pipeline run names the commit it built rather than the clock. Either form works locally, and the argument wins over the environment. -`ENV_FILE` is set as well as sourced, and the redundancy is deliberate. The script sources its own file regardless, so leaving `ENV_FILE` off would build and install against `secrets/.env` while the shell's `$HUGO_BASEURL` still named staging, and the run would check the staging site after publishing to the production root. The script prints the file it read, on every build, for that reason. +`ENV_FILE` is set as well as sourced, and the redundancy is deliberate. The script sources its own file regardless, so leaving `ENV_FILE` off would build and install against `secrets/local.production.env` while the shell's `$HUGO_BASEURL` still named staging, and the run would check the staging site after publishing to the production root. The script prints the file it read, on every build, for that reason. -It refuses to install a release that fails the build gate. `check-live-urls.sh` does take a base URL, which is where the sourced `$HUGO_BASEURL` goes. It follows all 1,245 URLs against the running mirror, checking each redirect's destination rather than trusting its status code. +It refuses to install a release that fails the build gate. `check-live-urls.sh` does take a base URL, which is where the sourced `$HUGO_BASEURL` goes. It follows every URL in the contract against the running mirror, checking each redirect's destination rather than trusting its status code. -Expect `PASS - 1245 URLs honored`. Anything less is a finding, and the output names each URL that failed and what it answered. +Expect a `PASS` naming the number of URLs honored, which is the two lists' combined length and grows as they do. Anything less is a finding, and the output names each URL that failed and what it answered. A documentation-only or workflow-only change does not need this. A change to the four paths above does, because for those CI's green is not evidence. @@ -104,10 +127,12 @@ A documentation-only or workflow-only change does not need this. A change to the Staging keeps Pangolin's authentication on, so an unauthenticated request never reaches the site. `check-live-urls.sh` presents a Pangolin resource access token when both halves of the pair are set, and sends nothing when neither is: ```sh -set -a; . secrets/staging.env; set +a +set -a; . secrets/vps.staging.env; set +a checks/check-live-urls.sh "$HUGO_BASEURL" ``` +The gate is the VPS staging environment's, so this is `secrets/vps.staging.env`. The local staging mirror sits behind Traefik on the maintainer's own network and carries neither half of the pair. + | Variable | Header | | --- | --- | | `PANGOLIN_ACCESS_TOKEN_ID` | `P-Access-Token-Id` | @@ -117,9 +142,9 @@ Set both or neither. Half a pair is a typo rather than a choice, and it is rejec Three properties of how the credential is handled, each there for a reason worth keeping: -- **It travels in a mode-`600` curl config file, not in `-H` arguments.** A command line is readable in `ps` for the life of the process, and this runs 1,245 of them. The config file is also the only form that survives the `export -f` the parallel checks run under, because bash cannot export an array. +- **It travels in a mode-`600` curl config file, not in `-H` arguments.** A command line is readable in `ps` for the life of the process, and this runs one per URL in the contract. The config file is also the only form that survives the `export -f` the parallel checks run under, because bash cannot export an array. - **It is sent to the base URL's own origin and nowhere else.** The check follows every redirect's destination, and every destination in the contract is same-origin today. A rule that one day points off-site must not mail the credential to whoever is on the other end. -- **A preflight request runs before the 1,245.** Behind an auth gate a wrong token fails *every* URL, and the output then reads as a site that has vanished rather than as a bad credential. The two are indistinguishable from the far end of a CI log, so the run stops on the first request with a message naming which of the two it was. +- **A preflight request runs before the rest.** Behind an auth gate a wrong token fails *every* URL, and the output then reads as a site that has vanished rather than as a bad credential. The two are indistinguishable from the far end of a CI log, so the run stops on the first request with a message naming which of the two it was. ## Deploying @@ -134,7 +159,7 @@ The deploy root and the base URL are the only host-specific values. A local run | Variable | Effect | | --- | --- | -| `ENV_FILE` | Which environment file to source. Defaults to `secrets/.env`. | +| `ENV_FILE` | Which environment file to source. Defaults to `secrets/local.production.env`. | | `DEPLOY_ROOT` | Fallback deploy root. The first argument wins. | | `HUGO_BASEURL` | Overrides the site base URL. | | `REQUIRE_BROTLI=1` | Fails rather than shipping gzip-only. CI sets this. | @@ -171,6 +196,49 @@ The script asserts both halves of that rather than assuming them. It fails when **Nothing prunes on the deploy path, and that is what keeps the deploy key's capability small.** A prune racing a deploy could take the rollback target, where a lingering release only costs disk. This is also why the key needs no delete capability, which is the property "Server Hardening" depends on. The count and the timer belong to the host, so this section records what the host declares rather than holding a second copy of it. See "Who Owns What". +## Log Review + +**Real traffic is the only source that finds what every check here is blind to.** The URL contract proves the URLs someone thought to list and the redirects derived from the export. It cannot know about a URL nobody recorded, because the lists are their own standard: the gates check the built site and the running server against those lists, never against the old platform that served the addresses. An address the crawl missed is therefore missing from every gate that reads them, and a visitor following a sixteen-year-old link is the one reader who tests for it. + +Review runs in both directions, which are the same two the media checks read and have the same blind spots for the same reason. + +| Direction | Question | Signal | Cadence | +| --- | --- | --- | --- | +| Outward | What did someone ask for that is not here? | non-200 responses | daily for the first week after cutover, then monthly | +| Inward | What is here that nobody has ever asked for? | URLs absent from every 200 | quarterly at the earliest, and a long tail by nature | + +**The outward pass is the one with an action.** A 404 on a path shaped like real content means the golden list missed a URL: add it to [`checks/golden-urls.txt`](./checks/golden-urls.txt) and add a redirect, per that file's own maintenance rules. Expect the raw counts to be dominated by scanners probing for `wp-login.php`, `.env`, and `.git/config`, which is noise from a site that used to run WordPress and should be filtered by shape rather than investigated. + +**The inward pass answers a question nothing else can.** Subtracting every URL that has ever returned 200 from the set the site builds names the content no reader has reached. It is slow evidence and deliberately so, since a post can go a year without a visit and still be worth keeping. Its first concrete use is the carried media that no page links and that the old platform never published, counted exactly by the parity gate and broken down in [`checks/README.md`](./checks/README.md): if nothing requests those files across a year, that settles whether carrying them is preservation or clutter, and no reasoning from the repository alone can settle it. + +### The log is three tiers, and each is blind to something + +A request crosses the proxy before it reaches the site, so no single log answers both questions. + +| Tier | Sees | Cannot see | +| --- | --- | --- | +| Traefik, or Pangolin's Traefik on the VPS | every request reaching the host, including unknown hostnames, TLS failures, and traffic aimed at names this site does not serve | which release answered, since Traefik logs request headers and not response headers | +| Pangolin, on the VPS only | requests the auth gate rejected | anything on the local mirrors, which have no gate | +| Caddy, per environment | path, status, and the `X-Blog-Release` that answered | anything the tiers above rejected, which never arrives | + +**A 404 count taken from Caddy alone is therefore a floor, not a total.** A request the edge refused is a reader who found nothing just as surely, and it appears in no Caddy log. Read the edge for what never arrived and Caddy for what arrived and failed, and treat the two as one answer. + +Two properties of the Caddy side are worth knowing before parsing it. Its access log is `format console`, so each line is a timestamp, a level, and a logger name followed by a JSON object rather than being JSON itself, and a parser that assumes one object per line reads nothing. And `trusted_proxies` is what makes `client_ip` the reader rather than the proxy, which is the same setting "Serving" describes as a security boundary. Without it every request in the log appears to come from one internal address, and the inward pass cannot distinguish a reader from a health check. + +### Retention Is the Prerequisite, and It Belongs to the Host + +**On the VPS the reviewable record is Traefik's access log**, at `/var/log/traefik/access.log`, one JSON object per line, one line per request, across every hostname the host serves. `RequestPath` carries the query string, so the legacy `/?p=` traffic is visible as itself. Request headers are dropped except `Referer` and `User-Agent`, which is what keeps the Pangolin resource access token out of a file that is retained and copied, and query strings are logged in full, so treat an extract as sensitive. + +**That log rotates and is eventually deleted, on a schedule the host sets and can change.** The window is long, and it is finite, so anything the inward pass depends on has to be copied off the host before the archive ages out. Read the current retention from the host rather than from this file, because a number written here is a number nothing checks. + +**Caddy's container log is the runtime log rather than the access log**, bounded by the container's own log rotation. It is where a failed config load and a dead `--watch` surface. It is not durable across a container recreate, since the Docker `json-file` log lives under the container id, and an operator editing the compose file is the event that discards it. A release deploy is not: an rsync and a symlink flip run no Docker operation at all. + +**Release attribution comes from a join rather than from a header.** Traefik cannot log a response header, so no access-log line names the release that answered. Join `StartUTC` against the release flip instead, which is exact outside a deploy window and ambiguous only inside one. + +**Retention on the local mirrors is a different question and is unsolved.** Those containers use Docker's `json-file` driver with the built-in defaults, so a mirror's log grows without bound and is discarded when its container is recreated. That matters less than it did on the VPS, because the mirrors serve no readers, and it belongs to the host rather than to this repository, the same split "Retention" and "Who Owns What" describe for release pruning. + +**The off-host copy of the access log is a design rather than a fact.** The VPS holds the only copy until the pull to the backup host runs, which makes the host's retention window the hard deadline rather than a comfortable margin. A review reads whatever survived and reports confidently on it either way, so settle this before the first review is expected to mean anything. + ## Who Owns What The site and the server it runs on are maintained separately, so the boundary is written down rather than inferred. This repo owns the artifact and what proves it correct; the host owns where a release may be written and what happens to it afterwards. @@ -245,7 +313,7 @@ The container reads three host paths, and only one of them a release ever writes Because it sits outside the bundle, no release updates it. Install or refresh it explicitly, once per environment, which is the same command against a different sourced file: ```sh -set -a; . secrets/.env; set +a # or secrets/staging.env +set -a; . secrets/local.production.env; set +a # or any other secrets/..env install -m 644 deploy/bootstrap.Caddyfile "$CADDY_APPDATA/config/Caddyfile" docker restart "$CADDY_CONTAINER" # only this file needs one, see below ``` @@ -260,7 +328,7 @@ Everything inside the bundle, `deploy/Caddyfile` and anything under `deploy/maps ## Redirects -The site answers 917 addresses it does not render, satisfied by 13 `redir` directives reading 5 map files, all inside the bundle. [`deploy/README.md`](./deploy/README.md) carries the per-class breakdown and the counts. This section covers the operational shape only, so the two do not restate each other. +The site answers far more addresses than it renders, satisfied by a small set of `redir` directives and the map files they read, all inside the bundle. [`deploy/README.md`](./deploy/README.md) carries the per-class breakdown and the counts. This section covers the operational shape only, so the two do not restate each other. Ordering is load-bearing, so every redirect lives in a single `route` block. Outside one, Caddy sorts directives by its own precedence rather than by file order, and the broad attachment rule claims the per-post comment feeds that the narrower rule must match first. diff --git a/README.md b/README.md index b25ad3b..10cb4dd 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,8 @@ See [Release History][history] for complete release notes and older versions. ## Table of Contents - [Use Cases](#use-cases) +- [Migration from WordPress](#migration-from-wordpress) +- [How a Change Reaches the Site](#how-a-change-reaches-the-site) - [Configuration](#configuration) - [Questions or Issues](#questions-or-issues) - [Development Environment Setup](#development-environment-setup) @@ -52,6 +54,97 @@ The site answers far more addresses than it renders pages, because it has served Deployment is a release directory plus a symlink. A build is installed alongside its predecessors, verified, and made live by swapping one link, so a rollback is the same swap in reverse. See [OPERATIONS.md][operations]. +## Migration from WordPress + +The site has served the same domain since 2008, across three platforms: Blogger until 2012, WordPress until 2026, and Hugo from then on. Converting the posts took an afternoon. Preserving sixteen years of inbound links was the work, and it is why this repository carries a URL contract and gates it rather than trusting the build. + +The account of that migration is a post on the site, [Moving This Blog From WordPress to Hugo][migration-post]. It covers what a WordPress export holds and what it leaves out, why the sitemap named barely a tenth of the addresses the site was actually serving, how the Blogger-era permalinks resolve through a lookup table rather than a pattern, why media fetched over HTTP is not the same bytes as the media in the export and only a content hash tells them apart, and which Hugo default moves every taxonomy archive to a new address without reporting anything. + +## How a Change Reaches the Site + +A change starts on a branch in this repository and ends as bytes on a VPS. Every stage carries a gate, and three of them add a human read, because the failure this repository exists to catch is an address that stops answering, and no check that skips a running server can see one. + +Both deploys are manual workflow dispatches, so a merge publishes nothing and each environment goes live by a deliberate run. + +[`WORKFLOW.md`][workflow] states the same CI machinery as a contract for tooling, and [`OPERATIONS.md`][operations] holds the procedure to run at a keyboard. + +```mermaid +flowchart LR + subgraph local["1. On a branch"] + direction TB + w["New branch, write the post"] --> b["Build, warnings are fatal"] + b --> p["Gate: every page that must render, renders"] + p --> m["Release onto the local mirror"] + m --> l["Gate: every URL in the contract, against a running server"] + l --> e1["Human: read the page"] + end + + subgraph gh["2. On GitHub"] + direction TB + pr["Pull request into develop"] --> ci["CI: lint, spelling, build, URL parity"] + ci --> rv["Automated review, then a human squash merge"] + end + + subgraph stg["3. Staging, behind the auth gate"] + direction TB + ds["Dispatch: deploy staging"] --> ls["Gate: every URL in the contract, run by CI"] + ls --> e2["Human: read staging"] + end + + subgraph prd["4. Production, public"] + direction TB + pm["Pull request: develop into main"] --> dp["Dispatch: deploy production"] + dp --> lp["Gate: every URL in the contract, run by CI"] + lp --> e3["Human: read production"] + end + + e1 --> pr + rv --> ds + e2 --> pm +``` + +**1. Write on a branch, and prove the artifact locally.** A post is a markdown file under `content/posts/`, written on a feature branch with whatever editor the author prefers. The build treats a warning as fatal, so a deprecated theme API fails it rather than accumulating. The build gate then checks the render half of the contract, which is every address that must return a page. + +Most of the contract is not pages, though. It is redirects, and a redirect is the web server's job, so no build reaches them. The release installs onto a mirror on the maintainer's own network, which runs the same Caddy container and the same bundle as the server behind the same Traefik front end. The live gate follows every URL in the contract against that mirror and checks each redirect's destination rather than its status code. A human then reads the page, because no gate has an opinion about the writing. + +**2. Open a pull request into `develop`.** CI runs lint, spelling, workflow and config validation, the build, and the render gate on a clean checkout. It does not prove the redirects, which is why the local run is a prerequisite rather than a convenience: a change to the Caddy config or to a redirect map goes green in CI while the redirect it broke stays broken. An automated review runs against the branch, and a human squash-merges it. + +**3. Deploy to staging.** A dispatched workflow builds the commit, uploads the release, flips the symlink, and runs the same live gate from CI against the running site. The local mirror proves the artifact, and staging proves the infrastructure that exists only on the server: routing, TLS, the deploy key, and its confined transport. Staging keeps its authentication gate on and the check presents a token, so a byte-identical copy of the public site is never exposed to a crawler. A human then reads it. + +**4. Promote, and deploy production.** `develop` merges into `main` through a pull request, and production is a second dispatch that accepts `main` alone. The same gate runs a third time, against the public address and with no token, and a human reads the result. + +Each branch feeds one environment: + +```mermaid +flowchart LR + feat["feature branch"] -->|squash| dev["develop"] + dev -->|merge commit| main["main"] + dev -.->|manual dispatch| stg["staging site"] + main -.->|manual dispatch| prod["production site"] +``` + +### What Happens While the Site Runs + +Publishing is half of it. The other half runs on its own cadence, because the contract proves only the addresses someone wrote down. + +```mermaid +flowchart LR + site["Production, serving"] --> log["Edge access log, one line per request"] + log --> outward["Outward: asked for, and not here"] + log --> inward["Inward: here, and never asked for"] + outward --> add["Add the address to the contract, add a redirect"] + add --> next["Ships with the next change, top of the pipeline"] + inward --> judge["Decide whether unread content is preservation or clutter"] + site --> cfg["Host configuration archived off the VPS"] + cfg --> rebuild["A rebuild restores the config and redeploys"] +``` + +**Real traffic is the only source that finds what the contract misses.** Every request to the host is recorded at the edge, one line per request, and the review runs in two directions. The outward pass reads non-200 responses. A 404 on a path shaped like real content means an old link nobody recorded, and the fix is to add the address to the contract and add a redirect, which then ships through the pipeline above like any other change. Scanners probing for `wp-login.php` dominate the raw count and are filtered by shape rather than investigated. The inward pass subtracts every address that has ever answered 200 from the set the site builds, which names content no reader has reached. That evidence accumulates slowly, and it is the only thing that settles whether media the old platform never published is worth carrying. + +**Backups protect the server rather than the site.** The site is reproducible from this repository by running a deploy, so what is worth keeping is the host's configuration: the container definitions, the proxy configuration, and the deploy account with its restricted key. A bare-metal restore rebuilds the host, restores that configuration, and deploys again. A procedure that backs up the deploy root protects a copy of something git already holds. + +[OPERATIONS.md][operations] holds the detail: the commands, the environments, the rollback, and the three log tiers with what each cannot see. + ## Configuration | Path | Holds | @@ -86,7 +179,7 @@ deploy/make-release.sh checks/check-live-urls.sh "$HUGO_BASEURL" ``` -The deploy root and the base URL come from an untracked file per environment under `secrets/`, copied from [deploy/env.example][env-example] and selected with `ENV_FILE`. The whole `secrets/` directory is gitignored, so host-specific values stay out of the published history. +The deploy root and the base URL come from an untracked file per environment under `secrets/`, named `..env`, copied from [deploy/env.example][env-example] and selected with `ENV_FILE`. `secrets/local.production.env` is the one read when `ENV_FILE` is unset. The whole `secrets/` directory is gitignored, so host-specific values stay out of the published history. ## 3rd Party Tools @@ -126,8 +219,10 @@ Licensed under the [MIT License][license]\ [history]: ./HISTORY.md [hugo-config]: ./hugo.yaml [license]: ./LICENSE +[migration-post]: ./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md [operations]: ./OPERATIONS.md [releases-link]: https://github.com/ptr727/Blog/releases +[workflow]: ./WORKFLOW.md diff --git a/TODO.md b/TODO.md index da499b2..e419543 100644 --- a/TODO.md +++ b/TODO.md @@ -8,7 +8,7 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y | Piece | State | | --- | --- | -| Content and media | done. 514 pages, 778 media files hash-verified against the export tar | +| Content and media | done. Carries every migrated post and page, with the media hash-verified against the export tar | | URL contract | done. 328 render, 917 redirect, 778 legacy image URLs, all gated | | Deploy shape | done. Proven on two local mirrors and on the VPS, by hand and by pipeline | | CI workflows | green. Validation runs on every pull request and feeds the required check | @@ -29,9 +29,14 @@ The site is built, gated in CI, and deployed to staging by pipeline. It is not y - **Retest the deploy transport against the real host**, which is [#33][issue-33] and is joint work with whoever holds the server. The transport now pins `StrictHostKeyChecking`, `UserKnownHostsFile`, and `BatchMode`, so it fails closed where it previously failed open, and a stale `DEPLOY_SSH_KNOWN_HOSTS` stops a deploy rather than being tolerated. Staging first, since a broken transport blocks the rollback path as well as the deploy. - **Declare what the VPS keeps.** `hugo.deploy.retention` asks for a retention count declared at the destination, and this repo's deploy credential is write-only by design so the prune belongs to the host. The ownership is recorded in `OPERATIONS.md`, the count is not, and the "ten releases" beside it describes `deploy/make-release.sh` on the local mirrors rather than the containers on the VPS. Confirm the host's timer and its count, then write it next to the ownership line. - **Prove a rollback through the pipeline.** A forced mid-deploy failure, then a flip back to the previous release, verified by `EXPECT_RELEASE` rather than by the transport exiting zero. The server side has been measured at well under a second by hand; what is unproven is that a **pipeline** run leaves the site serving when its deploy fails part way. -- **Deploy production once, to a name that is not the live one.** The production environment is configured and its Pangolin resource is deliberately disabled, so nothing has ever run against it. Validate there before the record moves. +- **Deploy production once, to a name that is not the live one.** The host side is done and the interim name is live: `blog.insanegenius.net` answers `200` unauthenticated, on a Let's Encrypt certificate issued 2026-08-07, serving `0000-placeholder`. The VPS agent calls this M7a, and M7b is the `.com` cutover. Three items belong to this repo before the first production deploy, all detailed in the VPS agent's §19 and §20. That file is not in the repository, so pull it first per [`OPERATIONS.md`](./OPERATIONS.md) "The Channel Between the Two Sides": + - **Set `HUGO_BASEURL` to `https://blog.insanegenius.net` on the `production` environment**, and back to `.com` at M7b. Hugo bakes `baseURL` into the canonical tags, the feed, and `sitemap.xml`, so a deploy without this edit serves a site whose every absolute URL points at the live WordPress site. It works mechanically and it is wrong. + - **Production emits `X-Robots-Tag: noindex, nofollow` for the length of the rehearsal**, deliberately, because `.net` serves a public duplicate of a live site and Certificate Transparency publishes the hostname. Where a check asserts `index, follow`, make the expected value a parameter rather than flipping a literal, since it reverts at M7b and a hardcoded literal is one more thing to remember at the wrong moment. + - **Answer the two questions in §19.3**: what `HUGO_BASEURL` holds on `production`, and whether anything else in the build or the checks hardcodes `blog.insanegenius.com`. +- **Decide what `robots.txt` says before the first production deploy.** Production answers 404 for it, so `X-Robots-Tag` is the only thing keeping the interim hostname out of an index. Crawlers are already asking: OAI-SearchBot, ClaudeBot, GPTBot, and Scrapy each arrived within three hours of the certificate being issued, at a hostname with no inbound links. - Lower the `blog` A-record TTL to 60s a day ahead, then flip it to the VPS, unproxied. -- Watch server logs for 404s daily for the first week, because real traffic finds what the golden list missed. Append anything new to `checks/golden-urls.txt` and add a redirect. +- **"Backed up" describes a design rather than a fact, for the log.** The VPS holds the only copy of the access log until the pull to the backup host runs, and that log rotates and ages out on a schedule the host owns, so the window is the deadline. The pull is two rsync lines the maintainer adds on the backup host, and the log stays deliberately outside the nightly encrypted archives because those are full copies with no dedupe. State that plainly in anything built on top of the log until the pull runs. +- Watch server logs for non-200s daily for the first week, then monthly, because real traffic finds what the golden list missed and the crawl that produced the list cannot. Append anything new to `checks/golden-urls.txt` and add a redirect. Read the edge as well as Caddy: a request the proxy refused never reaches the site's log, so a count taken from Caddy alone is a floor. The procedure, the three tiers and what each is blind to, and the inward pass that names content nobody has ever requested are in [`OPERATIONS.md`](./OPERATIONS.md) "Log Review". - Add the weekly non-blocking external-link-check workflow, which is the one gate that cannot be blocking because it fails on other people's outages. - Decommission WordPress.com only after **30 clean days**, and downgrade to free rather than deleting, which keeps the media reachable as a safety net and preserves the ability to re-export. Do not start sooner: the conversion fetched media over HTTP from the live site. @@ -66,12 +71,17 @@ Reconverged 2026-08-05, and the run is written up in [`reports/Blog/audit.md`](. **Measured against hub `main` `3b802b9eb9a841c0149d018f4db6ffa1b9419051`**, and the ref is named because `main` moves, which is the trap below. Every verbatim unit now matches: the re-vendor of `repo-config/configure.sh` this record previously owed, for the jq portability defect reported as [#549][issue-549] and fixed at the hub in [#553][pr-553], landed with this change. The links above are pinned to that same ref rather than to `main`, so this record stays checkable after the hub moves again. -Two findings are open at the hub. Neither is work this repo can do, and each changes what a fleet audit of this repo means, which is why they are recorded here rather than only in the issues. +Three findings are open at the hub, recorded here rather than only in the issues because each changes what a fleet audit of this repo means. The first two are not work this repo can do. The third is, once the change answering it is ground truth, and what it will ask is stated below the table. | Issue | What it means here | | --- | --- | | [#550][issue-550] | Nothing detects a repo missing from the registry, which is how this repo stayed invisible. Three other repos are still absent. | | [#552][issue-552] | The audit flags any carried `AGENTS.md` naming the template repo, and the byte-locked `Fleet Bootstrap` section names it. Carrying the canonical correctly cannot pass, and it is the one finding the current run cannot clear. | +| [#597][issue-597] | Filed from here, after a mandatory pre-merge gate in `OPERATIONS.md` was skipped on [#40][pr-40]. The ruling is that a verification a runner cannot perform needs a declared destination, not a better per-repo pointer. Answered by [#598][pr-598], which this repo owes work against once it is ground truth. | + +**What [#598][pr-598] will ask of this repo, once it is ground truth.** It is merged to the hub's `develop` and not to `main`, so it binds nothing yet, per the trap below about reading `main` as ground truth. It makes `Local Verification` a sixth declared `OPERATIONS.md` heading, leading the file as the only pre-merge one. This repo's `OPERATIONS.md` carries **13 level-two headings and matches none of the five declared today**, so the work is a rename and reorder rather than new prose: `Local Verification Before a Pull Request` becomes `Local Verification`, and `Backup and Restore` is one word from the declared `Backup and Recovery`. Both are near-misses rather than absences, which is the shape a heading check will mostly find in a repo that wrote its operational document before the spec declared headings. Do not start until the hub promotes it. + +No audit will report this meanwhile. `OPERATIONS.md` is presence-checked only, which is why the 2026-08-05 run above reported nothing about a file using none of the declared headings, and the heading check belongs to a hub cluster that has not shipped. Two more are resolved and are named because their absence from the table would otherwise read as an oversight. [#554][issue-554] made `spec/audit.py` report two DEFECTs here that no agent action could clear, and the fix is in the hub `main` this run measured against, so those two findings are gone. [#456][hub-issue] and [#558][hub-spec-issue] were the static-site type, now authored, per the section above. @@ -133,13 +143,14 @@ Secrets and variables, per environment. The App-token pair is repository-scoped `DEPLOY_SSH_PRIVATE_KEY` holds the same key in both environments, per the decision above. The environment split still carries the base URL, the SSH endpoint, and the staging-only token pair, so it is not decorative. -The deploy root is deliberately absent from this table. The rsync destination is anchored at the deploy key's confinement root, so the workflow names an environment rather than a host path, and a declared-but-unread name is drift no audit can tell from a missing one. The local `DEPLOY_ROOT` in `secrets/.env` is a different value and is still read. +The deploy root is deliberately absent from this table. The rsync destination is anchored at the deploy key's confinement root, so the workflow names an environment rather than a host path, and a declared-but-unread name is drift no audit can tell from a missing one. The local `DEPLOY_ROOT` in `secrets/..env` is a different value and is still read. [issue-33]: https://github.com/ptr727/Blog/issues/33 [migration-post]: ./content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md [pr-30]: https://github.com/ptr727/Blog/pull/30 +[pr-40]: https://github.com/ptr727/Blog/pull/40 @@ -153,4 +164,6 @@ The deploy root is deliberately absent from this table. The rsync destination is [issue-552]: https://github.com/ptr727/ProjectTemplate/issues/552 [issue-554]: https://github.com/ptr727/ProjectTemplate/issues/554 [issue-563]: https://github.com/ptr727/ProjectTemplate/issues/563 +[issue-597]: https://github.com/ptr727/ProjectTemplate/issues/597 [pr-553]: https://github.com/ptr727/ProjectTemplate/pull/553 +[pr-598]: https://github.com/ptr727/ProjectTemplate/pull/598 diff --git a/checks/README.md b/checks/README.md index bb15965..435916f 100644 --- a/checks/README.md +++ b/checks/README.md @@ -10,13 +10,15 @@ The contract is enforced by two gates, because one cannot cover both halves: | Gate | Proves | Runs | | --- | --- | --- | -| [`check-url-parity.py`](./check-url-parity.py) | Every URL that must render exists as a built page, every legacy image URL resolves, every local asset reference points at a real file | Against `public/`, in CI and before any release is installed | +| [`check-url-parity.py`](./check-url-parity.py) | Every URL that must render exists as a built page, every legacy image URL resolves, every local asset reference points at a real file, and the count of carried media linked from no page still equals its recorded baseline | Against `public/`, in CI and before any release is installed | | [`check-live-urls.sh`](./check-live-urls.sh) | Every redirect resolves, and its destination answers | Against a running server, which is the only thing that exercises a redirect | ## The two lists **`golden-urls.txt`, 328 URLs Hugo must render.** Missing any one is a hard CI failure. +**Every count on this page describes the two lists, not the site.** The lists are the legacy contract, closed by the migration, so a new post adds a URL the parity gate reports as `additional URLs built (not a failure)` and changes nothing here. A count moves only when a log review finds a legacy address the crawl missed, which is a deliberate append. + | Shape | Count | Note | | --- | --- | --- | | Tag archives | 180 | The site serves `/tag/`, and **Hugo defaults to `/tags/`** | @@ -28,7 +30,7 @@ The contract is enforced by two gates, because one cannot cover both halves: | Pages | 2 | `/about/`, `/viljoen-family/` | | Home | 1 | | -The 180 tag archives are exactly the tags carried by a published post. Three further terms answer with an empty page and are redirects rather than renders: `brultech` and `phyn`, which no published post uses, and `review`, an empty tag that is also a 12-post category. Hugo generates a term page only where posts exist, which is why the render list is 180 tags rather than 183. +The 180 tag archives are exactly the tags the migrated posts carry. Three further terms answer with an empty page and are redirects rather than renders: `brultech` and `phyn`, which no published post uses, and `review`, an empty tag that is also a 12-post category. Hugo generates a term page only where posts exist, which is why the render list is 180 tags rather than 183. **`redirect-urls.txt`, 917 URLs that must resolve but need not render.** These have no Hugo equivalent. Reproducing them would be absurd, and 404ing them discards real inbound links. @@ -39,7 +41,7 @@ The 180 tag archives are exactly the tags carried by a published post. Three fur | `?p=` shortlinks | 110 | Redirect to the permalink, via `p-ids.map` | | Attachment pages, root level | 107 | Redirect to the parent post, via `slugs.map` | | Per-post comment feeds | 107 | Redirect to the parent post | -| Date archives | 83 | Redirect to `/all/`, since **Hugo has no built-in year or month archive** | +| Date archives | 83 | Redirect to `/all/`, since **Hugo has no built-in year or month archive**. The matcher accepts any date, including dates absent from the list | | Blogger permalinks | 59 | Redirect to the current post, via `blogger.map` | | Blogger monthly archives | 21 | Redirect to `/all/` | | Author archive and pagination | 12 | Redirect to `/`, a single-author blog duplicating home | @@ -62,7 +64,7 @@ Two properties of the maps are non-obvious and easy to break when regenerating t **Adding a URL.** Real traffic finds what the lists missed. When a server log shows a 404 for an address that should work, append it to the appropriate list and add a redirect rule or map entry to cover it. The lists are append-only, per Directionality below. -**Regenerating the maps.** `build-redirects.py` rebuilds everything under `deploy/maps/` from the source export, which lives in a capture directory outside this repo and is passed as an argument. It is a provenance tool rather than a CI step, and it selects the export **by content**, failing unless exactly one candidate contains published posts. The capture holds both a full export and a media-only one with zero posts, and taking the wrong one yields empty maps that are indistinguishable from working ones until the redirects are live. +**Regenerating the maps.** `build-redirects.py` rebuilds everything under `deploy/maps/` from the source export, which lives in a capture directory outside this repo and is passed as an argument. **That directory's path is `CAPTURE_ROOT` in `secrets/local.production.env`**, and `OPERATIONS.md` "Rebuilding from the Exports" records what it holds and which parts of it a person can fetch again. It is a provenance tool rather than a CI step, and it selects the export **by content**, failing unless exactly one candidate contains published posts. The capture holds both a full export and a media-only one with zero posts, and taking the wrong one yields empty maps that are indistinguishable from working ones until the redirects are live. **Checking a count.** Every count above is derivable from the files, so check rather than trust: @@ -73,3 +75,30 @@ wc -l checks/golden-urls.txt checks/redirect-urls.txt checks/golden-media-legacy ## Directionality The parity check fails on a **missing** URL and only notes an **extra** one. New posts, new tags, and deeper pagination legitimately add URLs, and nothing legitimately removes a URL the site has served. That asymmetry is what makes the lists append-only, which in turn is what makes the length-floor assertion in `check-live-urls.sh` sound: without it a truncated list would make every assertion below it pass vacuously. + +Media is the one surface checked in **both** directions, and it has to be, because each direction is blind to the other's failure. Outward from a reference, the legacy list proves an inbound link still lands and the asset check proves a reference names a real file. Neither asks whether anything points at a given file, so an image dropped from a page during the conversion stays on disk, stays reachable at its own URL, and reports green in both directions while appearing nowhere on the site. The orphan check reads inward from the file and is the only one that sees it. + +Its constant is an exact count rather than a bound. A ceiling would let a drop leave slack behind for a later regression to hide in, so whatever lowers the count lowers the constant in the same change, and the check names the new number when it drops. + +A count is all the check can observe, and two causes reach each direction: it rises when a page stops linking media **or** when unlinked media is added, and it falls when media is linked from a page **or** when orphaned files are deleted. The messages name both, because naming one would send a reader looking for a page that never changed. + +**Both directions read absolute references as well as relative ones.** Hugo writes an absolute URL wherever a template resolves one against the base, which the entry-cover image on every list page does. Reading only rooted paths made those files look linked from nowhere while they were being displayed, and left a broken one unchecked in the other direction. The origin is read from the home page's canonical link rather than assumed, since staging and production build with different base URLs and a hardcoded host would check one environment's output against another's. No canonical link is a hard failure, because a guessed origin inflates the orphan count by exactly the pages that use one. + +## What the orphans are + +The count is not a backlog. It opened at 120 and was adjudicated against the captured live site under `blog-capture/mirror/`, which holds a crawl of the old platform including all 328 URLs the contract requires: + +| | | +| --- | --- | +| 17 | conversion losses, restored. The conversion emitted five `gallery` shortcodes empty | +| 5 | never orphans, referenced only by an absolute URL the check could not read | +| 97 | uploaded to the old platform's media library and never placed on a published page | +| 1 | that platform's site icon, superseded by the favicon set at the static root | + +**No image the old site served from its own uploads went unimported**, and no conversion loss remains. Anything that raises the count from here is therefore new, which is what makes the exact constant worth keeping. + +Three traps in that adjudication, each of which produced a wrong answer first and each cheap to re-trip: + +- **A regex cannot read nested elements.** The galleries are `wp-block-gallery` figures containing `wp-block-image` figures, and matching them by pattern reported a fictional 194-figure loss. An HTML parser gives the real number. +- **Only a surplus in the mirror is a finding.** A mirror page parsed as having *fewer* figures than the markdown means the parser missed that page's markup, never that images were lost. +- **The old platform generated an attachment page per upload, and a foreign host may also serve `/wp-content/uploads/`.** Counting attachment pages as places an image was displayed makes every unused upload look published, and matching an uploads path without checking its host attributes another site's file to this one. Both were hit here. diff --git a/checks/check-url-parity.py b/checks/check-url-parity.py index b3232f6..c2d4493 100755 --- a/checks/check-url-parity.py +++ b/checks/check-url-parity.py @@ -1,9 +1,10 @@ #!/usr/bin/env python3 """Verify the built site against the URL contract. -Checks that every URL which must render exists, that every legacy media URL resolves, and that -every local asset reference points at a file. Redirects need a running server and are checked -by check-live-urls.sh instead. +Checks that every URL which must render exists, that every legacy media URL resolves, that +every local asset reference points at a file, and that the number of carried media files +linked from no page still equals its recorded baseline. Redirects need a running server and +are checked by check-live-urls.sh instead. """ import pathlib @@ -21,6 +22,21 @@ CHECKS = pathlib.Path(__file__).resolve().parent +# Media carried by the import that no built page links to. +# The other two media checks run outward from a reference and cannot see these: a legacy URL +# resolving proves an inbound link still lands, and a reference resolving proves it names a real +# file. Neither asks whether anything points at a given file, so an image the conversion dropped +# from a page stays reachable by URL, invisible on the site, and green in both directions. +# It opened at 120, of which 17 were conversion losses restored from the captured live site and 5 +# were never orphans at all, being referenced only by an absolute URL this check could not read. +# The 98 that remain are adjudicated rather than unknown: 97 were uploaded to the old platform's +# media library and never placed on any published page, and one is that platform's site icon, +# superseded by the favicon set at the static root. Nothing here is a conversion loss, and no image +# the old site served from its own uploads went unimported. checks/README.md carries the method. +# The count is exact rather than a bound, so whatever lowers it lowers this in the same change and +# slack can never accumulate for a later regression to hide in. +ORPHANED_MEDIA = 98 + def load(name): lines = [ln.strip() for ln in (CHECKS / name).read_text().splitlines()] @@ -69,30 +85,110 @@ def check_media(public): return missing -def check_assets(public): - """Check that every local asset a built page references exists on disk. +def site_origin(public): + """The site's own scheme and host, read from the artifact rather than assumed. - Catches a media file renamed, dropped, or never localized. + Staging and production build with different base URLs, so a hardcoded host would check + one environment's output against another's and silently match nothing. + """ + home = public / "index.html" + if not home.is_file(): + sys.exit(f"FAIL: {home} is missing - run hugo first") + text = home.read_text(encoding="utf-8", errors="ignore") + found = re.search(r'rel=["\']?canonical["\']?\s+href=["\']?(https?://[^/"\'>\s]+)', text) + if not found: + # Without the origin, every absolute reference reads as external and the orphan count + # inflates by exactly the pages that use one. Guessing would be worse than stopping. + sys.exit("FAIL: no canonical link on the home page - cannot determine the site's own origin") + return found.group(1) + + +def collect_refs(public): + """Every local asset reference in the built pages. + + Read once and shared, since the assets and orphans checks are the same reference set + read in opposite directions. """ # Minification drops the quotes around an attribute value that does not need them. # Matching only the quoted form checks a fraction of the references and calls it a pass. quoted = re.compile(r'(?:src|href|srcset)="(/(?:media|external)/[^"]+)"') bare = re.compile(r"(?:src|href|srcset)=(/(?:media|external)/[^\s\"'>]+)") - refs, missing = set(), [] + # Hugo writes an absolute URL wherever a template resolves one against the base, which the + # entry-cover images on every list page do. Read as external, those files look linked from + # nowhere while being displayed, and a broken one is never checked at all. + absolute = re.compile(re.escape(site_origin(public)) + r'(/(?:media|external)/[^\s"\'>]+)') + refs = set() for page in public.rglob("*.html"): text = page.read_text(encoding="utf-8", errors="ignore") refs.update(quoted.findall(text)) refs.update(bare.findall(text)) - for ref in sorted(refs): - # Imported references carry resize parameters a static file server ignores. - # Some also escape an underscore, which a server decodes before looking up the file. - path = unquote(ref.split("?", 1)[0].split("#", 1)[0]) - if not (public / path.lstrip("/")).is_file(): - missing.append(ref) + refs.update(absolute.findall(text)) + return refs + + +def ref_to_path(ref): + """Map a reference to the path under the built site it names.""" + # Imported references carry resize parameters a static file server ignores. + # Some also escape an underscore, which a server decodes before looking up the file. + return unquote(ref.split("?", 1)[0].split("#", 1)[0]).lstrip("/") + + +def check_assets(public, refs): + """Check that every local asset a built page references exists on disk. + + Catches a media file renamed, dropped, or never localized. + """ + missing = [ref for ref in sorted(refs) if not (public / ref_to_path(ref)).is_file()] print(f"assets : {len(refs) - len(missing)}/{len(refs)} local asset references resolve") return missing +def check_orphans(public, refs): + """Check that every carried media file is linked from some built page. + + The reverse of the assets check, and the only one that can see an image the conversion + dropped from a page: it stays on disk and reachable by URL, so nothing else objects. + """ + linked = {ref_to_path(ref) for ref in refs} + carried, orphaned = 0, [] + for tree in ("media", "external"): + root = public / tree + if not root.is_dir(): + continue + for path in root.rglob("*"): + if not path.is_file(): + continue + carried += 1 + # `linked` holds URL paths, which are always forward-slashed, so a native separator + # here would match nothing and report every carried file as an orphan. check_render + # normalizes for the same reason. + rel = str(path.relative_to(public)).replace("\\", "/") + if rel not in linked: + orphaned.append(rel) + orphaned.sort() + # No media at all is a broken build, not progress. Left to the comparison below it reads as + # zero orphans, which is fewer than the baseline, and the advice would be to lower + # ORPHANED_MEDIA to 0 - a gate talking the reader into switching it off. + if carried == 0: + print("orphans: no media files in the built site - the output is incomplete or mislocated") + return ["public/media and public/external are both absent or empty"] + print(f"orphans: {len(orphaned)} of {carried} carried media files are linked from no page") + if len(orphaned) == ORPHANED_MEDIA: + return [] + # The explanation is printed rather than returned, so the caller's count stays the orphan + # count. A diagnostic carried in the failure list would make the reported total one too many. + # A count is all this can observe, and two causes reach each direction. Naming one of them + # would send a reader looking for a page that never changed. + if len(orphaned) > ORPHANED_MEDIA: + print(f" expected {ORPHANED_MEDIA} - a page stopped linking media, or unlinked media was added") + return orphaned + print( + f" expected {ORPHANED_MEDIA} - media was linked from a page, or orphaned files were " + f"removed; lower ORPHANED_MEDIA to {len(orphaned)} in this change rather than leaving the slack" + ) + return orphaned + + def main(argv): if len(argv) != 2: sys.exit(f"usage: {argv[0]} ") @@ -100,11 +196,13 @@ def main(argv): if not public.is_dir(): sys.exit(f"FAIL: {public} is not a directory - run hugo first") + refs = collect_refs(public) failures = [] for label, missing in ( ("render", check_render(public)), ("media", check_media(public)), - ("assets", check_assets(public)), + ("assets", check_assets(public, refs)), + ("orphans", check_orphans(public, refs)), ): if missing: failures.append((label, missing)) diff --git a/content/posts/2021/08/09/esp32-water-and-gas-utility-meter.md b/content/posts/2021/08/09/esp32-water-and-gas-utility-meter.md index e89b580..1e41c40 100644 --- a/content/posts/2021/08/09/esp32-water-and-gas-utility-meter.md +++ b/content/posts/2021/08/09/esp32-water-and-gas-utility-meter.md @@ -23,6 +23,13 @@ Below are pictures of the finished case and the utility meters: {{< gallery cols="4" >}} +{{< figure src="/media/2018/08/20180811_180521052_ios.jpg?w=1024" alt="" caption="" >}} + +{{< figure src="/media/2020/08/img_6638.jpg?w=768" alt="" caption="" >}} + +{{< figure src="/media/2021/08/img_8178_1.jpg?w=400" alt="" caption="" >}} + +{{< figure src="/media/2021/08/img_8178.jpg?w=569" alt="" caption="" >}} {{< /gallery >}} The ESPHome code uses the [pulse\_meter](https://esphome.io/components/sensor/pulse_meter.html) sensor, with the total sensor being measured by the HA [utility\_meter](https://www.home-assistant.io/integrations/utility_meter/) integration. The sensor produces flow rate measurements and incrementing absolute consumption, and it is the consumption, not flow rate, that is needed for the utility meter integration. diff --git a/content/posts/2022/10/30/installing-flair-smart-vents-to-keep-room-temperatures-balanced.md b/content/posts/2022/10/30/installing-flair-smart-vents-to-keep-room-temperatures-balanced.md index 4a52263..93d29bd 100644 --- a/content/posts/2022/10/30/installing-flair-smart-vents-to-keep-room-temperatures-balanced.md +++ b/content/posts/2022/10/30/installing-flair-smart-vents-to-keep-room-temperatures-balanced.md @@ -57,6 +57,13 @@ The look and feel of the Puck was a bit disappointing for the price, small non-t {{< gallery cols="4" >}} +{{< figure src="/media/2022/10/img_1646.jpg?w=1024" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1647.jpg?w=594" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1655.jpg?w=665" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1657.jpg?w=426" alt="" caption="" >}}FLAIR puck vs. my old Ecobee3 {{< /gallery >}} I installed the iOS app, set the puck in gateway only mode, followed the installation steps in the app, including creating an account, connecting the puck to WiFi, and connecting Flair to Ecobee for access to my thermostats. I installed batteries in a vent, without installing the actual vent but observing its behavior, and added the vent to Flair using the iOS app by entering pairing mode, no problems. @@ -94,6 +101,13 @@ I ended up removing the old vents, and visited my paint store several times to c {{< /gallery >}} {{< gallery cols="5" >}} +{{< figure src="/media/2022/10/img_1651.jpg?w=663" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1663-edited.jpg" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1665-edited-1.jpg" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1668.jpg?w=589" alt="" caption="" >}}FLAIR vent is smaller than standard vents {{< /gallery >}} With all the trouble caused by the too narrow vents, I wondered if there was a standard to sizes, and if so, if Flair was following it. I researched vent sizes across several HVAC supply sites and hardware stores, and my conclusion is that the Flair [vents](https://www.flair.co/smart-vent-engineering-and-dimensional-data-public) are always longer and narrower than standard wall/ceiling vent sizes (I did not compare floor vent sizes). @@ -152,6 +166,13 @@ I looked for vendors with reasonable pricing and thin white paintable plastic ma {{< gallery cols="4" >}} +{{< figure src="/media/2022/10/img_1708.jpg?w=726" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1709.jpg?w=617" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1752.jpg?w=601" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1751.jpg?w=519" alt="" caption="" >}}Laser cut white acrylic flange {{< /gallery >}} I shared the Fusion 360 projects and DXF files in [Autodesk Gallery](https://gallery.autodesk.com/projects/173188/flair-vent-flanges), and I created a shared [Ponoko cart](https://www.ponoko.com/share/MgBrJSraTWp83F79ob_Ql) with a ready to order 6x10 and 6x12 flange. For sizes other than 6x10 or 6x12 you will need to use Fusion 360 (free for hobby use) and adjust the dimensions in the sketch (\[modify\], \[change parameters\]), then export the projection as a DXF file. diff --git a/content/posts/2022/10/31/halloween-pumpkins-and-lights.md b/content/posts/2022/10/31/halloween-pumpkins-and-lights.md index 1c3b109..cf2ba0c 100644 --- a/content/posts/2022/10/31/halloween-pumpkins-and-lights.md +++ b/content/posts/2022/10/31/halloween-pumpkins-and-lights.md @@ -26,6 +26,11 @@ Last year I used wood planks as backing on my coro props, but that made hanging {{< gallery cols="4" >}} +{{< figure src="/media/2022/10/img_1731.jpg?w=768" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1702.jpg?w=1024" alt="" caption="" >}} + +{{< figure src="/media/2022/10/img_1700.jpg?w=623" alt="" caption="" >}} {{< /gallery >}} This year I used a [Kulp K16A-B](https://kulplights.com/product/k16a-b/) controller and a [Wired Watts](https://www.wiredwatts.com/build-a-controller-kit) enclosure that saved me the effort of 3D printing my own mounting hardware, and I just used more ports vs. power injecting. I also added [Wally's Lights 30W RGB floods](https://wallyslights.com/collections/flood-lights/products/30-watt-rgbw-12v-flood-light), and [Mattos Designs Diamond Pixels Balls](https://mattosdesigns.com/collections/smart-pixels/products/pre-buy-3-md-12v-diamond-3d-pixel-ball-36ct) wrapped around the tree. diff --git a/content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md b/content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md index cec4a88..e915a59 100644 --- a/content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md +++ b/content/posts/2026/08/01/moving-this-blog-from-wordpress-to-hugo.md @@ -57,6 +57,26 @@ I ended up splitting the contract in two: **328 URLs that Hugo must render**, an The order matters more than the tools. Everything up to step 4 is reversible and costs nothing. Do all of it before you convert anything. +### 0. Decide where the inputs live, before you fetch any of them + +Every step below produces a file you will still be reading months later. The crawl is the input to every check. The export is the input to every redirect map. The media tar is the only correct copy of your images. Put all of it in one directory, outside the site repository, and keep it there. + +Outside the repository, because none of it belongs in a published history: it carries commenter email addresses, IP addresses, and a complete copy of a site you are about to take down. In one directory, because the tooling that derives your URL contract has to be pointed at it, and a derivation you cannot re-run is a result nobody can check. + +The shape that worked here: + +```text +capture/ + export/raw/ the content export, as downloaded + export/media-tar/ the media export, unpacked + mirror/ the crawl, as the old site served it + inventory/ the URL and media lists derived from that crawl +``` + +Then record where that directory is, in the repository, alongside whatever other machine-specific values you keep out of git. That sounds too obvious to write down, which is exactly why it does not get written down: the one thing you cannot re-derive from the repository is the location of the thing everything was derived from. + +Two of those four are re-fetchable from WordPress for as long as the account exists, and those two are the only things a person has to go and download. The crawl and the inventories are not re-fetchable at all, and they stop being possible the day the old hosting ends. + ### 1. Capture the URL surface while the old site is still up Do this **first**. Once you cancel the old hosting you can never reconstruct this, and it is the input to every check later. diff --git a/deploy/README.md b/deploy/README.md index 4d7759f..43124fb 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -57,23 +57,34 @@ checks/check-live-urls.sh "$HUGO_BASEURL" ``` The deploy root and the base URL are the only host-specific values, and they pair per -environment. Copy [`env.example`](./env.example) to `secrets/.env`, which is the file read when -`ENV_FILE` is unset, and add `secrets/.env` for each further environment. A single -environment therefore needs `secrets/.env` and nothing else, since a differently named file is -read only when `ENV_FILE` names it. `secrets/` is gitignored as a whole directory, so a value -naming one machine cannot reach a public repo by being added to a file nobody remembered to -ignore. CI passes them explicitly instead, which keeps a pipeline run self-describing: +environment. Copy [`env.example`](./env.example) to `secrets/local.production.env`, which is the +file read when `ENV_FILE` is unset, and add `secrets/..env` for each further +environment. A single environment therefore needs `secrets/local.production.env` and nothing +else, since a differently named file is read only when `ENV_FILE` names it. `secrets/` is +gitignored as a whole directory, so a value naming one machine cannot reach a public repo by +being added to a file nobody remembered to ignore. CI passes them explicitly instead, which +keeps a pipeline run self-describing: ```sh HUGO_BASEURL= deploy/make-release.sh "$(git rev-parse --short HEAD)" ``` -**One file per environment, selected by `ENV_FILE`.** `secrets/.env` is the default and is read -when `ENV_FILE` is unset, so a single-environment host needs nothing else: +**One file per environment, named `secrets/..env`, selected by `ENV_FILE`.** +Both halves are spelled out, so a name says which machine it describes as well as which +environment on it, and the four in the fleet read as one set. `secrets/local.production.env` is +the default and is read when `ENV_FILE` is unset, so a single-environment host needs nothing +else: ```sh -deploy/make-release.sh # secrets/.env -ENV_FILE=secrets/staging.env deploy/make-release.sh # the staging site on the same host +deploy/make-release.sh # secrets/local.production.env +ENV_FILE=secrets/local.staging.env deploy/make-release.sh # the staging site on the same host +``` + +A file whose `DEPLOY_SSH_HOST` is set describes a root on another machine, so this script refuses +to create that path here and asks for a local one to assemble a bundle into: + +```sh +ENV_FILE=secrets/vps.staging.env deploy/make-release.sh /path/to/bundle ``` Selecting the file is the only way to switch environments. The file is sourced with `set -a`, @@ -93,7 +104,7 @@ every build for that reason. | Variable | Effect | | --- | --- | -| `ENV_FILE` | Which environment file to source. Defaults to `secrets/.env`. | +| `ENV_FILE` | Which environment file to source. Defaults to `secrets/local.production.env`. | | `DEPLOY_ROOT` | Fallback deploy root. The first argument wins. | | `HUGO_BASEURL` | Overrides the site base URL. Hugo maps `HUGO_` onto config natively. | | `REQUIRE_BROTLI=1` | Fails rather than shipping gzip-only. CI sets this. | @@ -109,7 +120,7 @@ open the auth gate: Set both or neither; half a pair is rejected as the typo it is. They go to curl through a mode-`600` config file rather than as `-H` arguments, which keeps the credential out of the -`ps` output of 1,245 requests, and is also the only form that survives the `export -f` the +`ps` output of one request per URL in the contract, and is also the only form that survives the `export -f` the parallel checks run under. The token is sent to the base URL's own origin and to nothing else, so a redirect that one day points off-site cannot carry it away. @@ -203,7 +214,7 @@ So the default is the value that is harmless on production, and `noindex` is rea asking for it explicitly. `checks/check-live-urls.sh` asserts this when `EXPECT_SITE_ENV` is set, before it checks the -1,245 URLs, since checking the contract against the wrong environment proves nothing. +contract, since checking the contract against the wrong environment proves nothing. ## Trusting the proxy @@ -259,6 +270,8 @@ Directives and URL classes are not one to one, in both directions. `@mapped` is Each row below is a **URL class**, named by the matcher that serves it, so the table can be checked against [`Caddyfile`](./Caddyfile) by grep rather than by trust. +Every class below is a legacy shape, closed by the migration, so no count here moves when content is added. New content is served by the render half of the contract, never by these. + | Matcher | Class size | Shape | | --- | --- | --- | | `@post_child` | 216 | `/YYYY/MM/DD/post//` -> the post, attachment pages | @@ -266,7 +279,7 @@ Each row below is a **URL class**, named by the matcher that serves it, so the t | `@post_id` | 110 | `/?p=` -> the permalink, via `p-ids.map` | | `@post_child_feed` | 107 | `/YYYY/MM/DD/post//feed/` -> the post, ordered **before** `@post_child` | | `@mapped` via `slugs.map` | 107 | bare `//` -> best destination | -| `@date_archive` | 83 | `/YYYY/`, `/YYYY/MM/`, and their pagination -> `/all/` | +| `@date_archive` | 83 | `/YYYY/`, `/YYYY/MM/`, and their pagination -> `/all/`. The matcher accepts any date, including dates absent from the list | | `@mapped` via `blogger.map` | 59 | `/YYYY/MM/slug.html` -> the current post | | `@blogger_archive` | 21 | `/YYYY_MM_01_archive.html` -> `/all/`, any date, including ones never covered | | `@author` | 12 | `/author//`, its pagination and feed -> `/` | @@ -275,9 +288,9 @@ Each row below is a **URL class**, named by the matcher that serves it, so the t | `@blogger_feed` | 2 | `/feeds/posts/default` -> `/feed.xml`, Blogger's Atom feed | | `@blogger_page` | 2 | `/p/.html` -> `//`, Blogger's static-page shape | -**Those thirteen classes sum to 917**, which is the line count of [`checks/redirect-urls.txt`](../checks/redirect-urls.txt) and the whole redirect contract. +**Those thirteen classes account for every line in [`checks/redirect-urls.txt`](../checks/redirect-urls.txt), with nothing in the contract outside the table.** Completeness is the property worth holding, and the list's line count is how to check it. -`@uploads` is deliberately absent from that table and from the 917. It rewrites `/wp-content/uploads/(.*)` to `/media/$1`, preserving all 778 legacy image URLs, which are gated by `golden-media-legacy.txt` on their own. Counting them here would double-count a set that has its own list. +`@uploads` is deliberately absent from that table and from the redirect contract. It rewrites `/wp-content/uploads/(.*)` to `/media/$1`, preserving all 778 legacy image URLs, which are gated by `golden-media-legacy.txt` on their own. Counting them here would double-count a set that has its own list. `@label` is the fourteenth class and is deliberately **not** in the contract. `/search/label/