Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,15 @@ Two properties of the Caddy side are worth knowing before parsing it. Its access

The outward pass is four filters over the edge log, and each one exists because skipping it produced a wrong answer once.

**Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. Filter on user agent: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0` and the 711 that remained are the entire real dataset. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large.
**Exclude this repository's own deploy gate first.** `check-live-urls.sh` requests the whole URL contract on every deploy, so an unfiltered day is mostly a recording of our own `curl`. A count that omits this step is measuring the pipeline rather than the readers, and it will be an order of magnitude too large.

The mechanism is the `X-Blog-Check` request header, which the scripted checks send on every request they make, so `jq 'select(.["request_X-Blog-Check"] == null)'` is the filter. It is only as complete as the tagging is, which is the first bullet below. Its value is a source and an id rather than a boolean, so a run is identifiable rather than merely excludable, and real values look like `github/31322640628-1` from CI, `vps/smoke` from the host side, and `proxmox/media-dev` from here. The shape is enforced by `check-live-urls.sh`, which takes exactly one `/` and only letters, digits, `.`, `_`, `-`, so a placeholder written with angle brackets is a description rather than something to paste.

- **The field exists only because the edge is configured to log that header**, which is the host side's to hold and not this repository's. An absent field therefore has two meanings, an untagged request or a capture that stopped, and they are not distinguishable from the log alone. Confirm the capture is live before reading a day's absence as a day of real traffic.
Comment thread
ptr727 marked this conversation as resolved.

- **A hand probe carries it only because whoever runs it adds it.** `check-live-urls.sh` sends it on every request and a bare `curl` sends nothing, so an interactive probe passes `-H "X-Blog-Check: proxmox/media-dev"`. The source half stays `proxmox`, which is where the probe came from, and the id half is where the purpose goes. Two untagged probes turned up against 3,100 tagged ones in the 2026-08-09 deploy window.
- **Absence is not proof of a human**, since a scanner sends no header either, so this pairs with the scanner-shape filters below rather than replacing them. The field is forgeable and must never reach auth, rate limiting, robots handling, or caching.
- **Before 2026-08-09 the log carries no such field**, and user agent is the only key for those days: on 2026-08-08, 9,285 of 9,996 requests were `curl/8.5.0`, leaving 711 real ones. That key is a coincidence rather than a rule, since the CI runner's curl and the host's are byte-identical and only the rotating client address separates them, which is why the header exists.

**A referer does not implicate this site unless it points somewhere else.** The rule worth applying is that a 404 carrying a referer is a broken link and a 404 without one is a typed or probed address, and it fails on scanners, which set `Referer` to the request URL itself. Every one of the 36 referer-bearing site-host 404s on 2026-08-08 was self-referential, so the unrefined rule reported three dozen broken links on a site that had none. Discard the matches before counting, and **normalize the scheme rather than comparing it**, because a scanner reaching an HTTPS site routinely sends an `http://` referer for the same address. Comparing against the request's own scheme therefore matches nothing and leaves every false positive in place: on the 2026-08-08 data the naive form kept all 36 where the normalized form kept none.

Expand All @@ -291,7 +299,7 @@ Widen `== 404` to `>= 400` for the whole non-200 sweep the table above describes

### Retention Is the Prerequisite, and It Belongs to the Host

**On the VPS the reviewable record is Traefik's access log**, at `/var/log/traefik/access.log`, one JSON object per line, one line per request, across every hostname the host serves. `RequestPath` carries the query string, so the legacy `/?p=<id>` traffic is visible as itself. Request headers are dropped except `Referer` and `User-Agent`, which is what keeps the Pangolin resource access token out of a file that is retained and copied, and query strings are logged in full, so treat an extract as sensitive.
**On the VPS the reviewable record is Traefik's access log**, at `/var/log/traefik/access.log`, one JSON object per line, one line per request, across every hostname the host serves. `RequestPath` carries the query string, so the legacy `/?p=<id>` traffic is visible as itself. The log records only three request headers, which appear as the keys `request_Referer`, `request_User-Agent`, and `request_X-Blog-Check`, and omits the rest from the record rather than stripping them from the request, which still arrives intact. That omission is what keeps the Pangolin resource access token out of a file that is retained and copied, and query strings are logged in full, so treat an extract as sensitive. That keep-list is the host's to hold and is the precondition the outward pass depends on: a header absent from it does not appear in the log at all, which is indistinguishable from a request that never sent one.

**That log rotates and is eventually deleted, on a schedule the host sets and can change.** The window is long, and it is finite, so anything the inward pass depends on has to be copied off the host before the archive ages out. Read the current retention from the host rather than from this file, because a number written here is a number nothing checks.

Expand Down
Loading