Pull the VPS access log off-host, and schedule it: the leg is written but not installed

[ptr727]

TODO.md records that the off-host copy of the access log "describes a design rather than a fact", and that the pull is two rsync lines the maintainer adds on the backup host. Those lines are now written, tested against the live VPS, and not installed — every install step needs root.

This is host tooling rather than repo content, so nothing here is a change to this repository. It is filed here because this repo is what depends on it: OPERATIONS.md "Log Review" is unrunnable past the VPS's retention window until it lands.

What exists now

/usr/local/bin/vps-backup-pull on Proxmox already pulls the encrypted archives and the plaintext host config, and is already parameterised by VPS_HOST, DEST, REMOTE_ARCHIVE_DIR and SSH_OPTS. It has no log leg and no schedule.

A patched copy adds the log leg in the same style, with both ends as environment variables:

Variable	Default	What
REMOTE_TRAEFIK_LOG_DIR	/var/log/traefik/archive	rotated edge access logs, on the VPS
REMOTE_BLOG_LOG_DIR	/var/log/blog/legacy	one-off Caddy container dumps, on the VPS
LOG_DEST	/data/backup/vps-logs	where both land, on Proxmox

Plus a --no-logs flag mirroring the existing --no-hostconfig, and a systemd service and timer for the schedule that has never existed.

Verified by running it: 35/35 archives checksum-verified, 33 host config files, 4 log files landed. bash -n clean, and shellcheck reports only the two informational findings the original already had.

Three decisions, each commented at its point in the script

• --delete never reaches the log leg, whatever is passed. The VPS prunes at 400 days by design and this host is the copy that outlives it, so mirroring that prune would delete the only remaining copy at exactly the moment it became the only one.
• Today's live access.log is not fetched. It is still being appended to, so a copy is a torn prefix the next run refetches; rotation at 00:00 UTC is what makes a file eligible. Read the live file over SSH when analysing today.
• LOG_DEST is separate from DEST, and mode 700. Query strings are logged in full, so the tree is as sensitive as the log, and its retention requirement is the opposite of the archives'.

The timer runs 09:00 UTC daily, behind both producers on the VPS rather than beside them: logrotate rotates at 00:00 UTC and pangolin-backup.timer writes the day's archive at 08:03 UTC. Pulling before the archive exists fetches yesterday's and reports success, which is the failure that looks like a working backup. Persistent=true covers a host that was off.

What is left, and who has to do it

1. Install — needs root: the script to /usr/local/bin/, the two units to /etc/systemd/system/, then systemctl enable --now vps-backup-pull.timer.
2. Send the change back to the VPS. The script's own header says the canonical copy lives on the VPS at /usr/local/share/pangolin-maint/vps-pull.sh and that the Proxmox copy is deployed from there, so a change made only on Proxmox is reverted by the next deploy of that header's scp. Better routed through the VPS agent, which owns that file.
3. Decide who prunes the local archive set. --delete is deliberately absent from the unit, so the Proxmox copy grows past the VPS's 14-archive window — it is already 38 here against 35 there. That is the copy doing its job rather than drift, but nothing prunes it and it has no owner.

Until step 1 lands, the 400 days on the VPS remain the only copy of the access log, and anything built on that log should say so.

[ptr727]

Installed on Proxmox 2026-08-08. Steps 1 and 2 of the three below are now done; the third is unchanged.

/usr/local/bin/vps-backup-pull              root:root 755
/etc/systemd/system/vps-backup-pull.service root:root 644
/etc/systemd/system/vps-backup-pull.timer   root:root 644  enabled, active, next 09:00 UTC daily

Run twice under systemd, not just from a shell, Result=success ExecMainStatus=0 both times. The previous script is kept beside it as vps-backup-pull.bak-20260808.

One defect found by installing it, and it was pre-existing

The first systemd run left a journal reading Starting… / Finished and nothing else. The cause: the closing summary went through say(), which --quiet silences, so --quiet printed nothing while the usage text promised "errors and the summary only". That is worst exactly where --quiet is used — under a timer — because a successful scheduled backup then leaves no evidence that it copied anything.

The summary now prints unconditionally and states the counts, so one journal line is the audit record:

=== done -- 42 archive(s), 38 verified, 468M in /data/backup/vps; 4 log file(s), 1.1M in /data/backup/vps-logs

What that line already shows

42 archives here against 38 on the VPS. The four extra are ones the VPS has since pruned from its fourteen-archive window and this host still holds, which is the copy doing its job. It is also the third item below, still unowned: nothing prunes the local set.

Still open

Send the change back to the VPS. The script's own header says the canonical copy is at /usr/local/share/pangolin-maint/vps-pull.sh on the VPS and that the Proxmox copy is deployed from there, so this change is reverted by the next deploy of that header's scp. Not done here: writing to that host is the VPS side's call, and folding it in there is better than copying it back, since that side owns the file and the change references its pangolin-backup conventions.

[ptr727]

Done, and the ownership question this raised is decided.

What landed

The log leg, both units, and the schedule are installed and running on the backup host, and the script is no longer host-only tooling: it is committed at ops/ with an installer, a drop-in template, and a README, merged in #62.

The variable names in the issue body above are superseded. LOG_DEST is LOG_ARCHIVE_ROOT, DEST is BACKUP_ARCHIVE_ROOT, VPS_HOST is VPS_SSH_HOST, and REMOTE_TRAEFIK_LOG_DIR is VPS_TRAEFIK_LOG_ARCHIVE. They were unified with the names this repository's environment file already used, so the side that writes the off-host copy and the side that reads it cannot drift. Every value is described once in ENVIRONMENT.md, and checks/check-env-docs.py fails CI if one is added without a description.

Installed and verified rather than assumed: the running script is byte-identical to the committed one, systemd resolves the account and the mount requirement from the drop-in, and the environment file is 600 root:root.

The question this issue implied, now answered

The script existed only on the backup host, and a copy on the VPS at /usr/local/share/pangolin-maint/vps-pull.sh claimed to be canonical. Two hosts each claiming to be the source is what let them diverge in both directions, so the header on one of them instructed an scp that would have deleted the access-log leg outright.

The host side has ruled: the git tree is canonical. Its copy is relabelled a recovery snapshot, the destructive instruction is replaced with a banner saying the arrow only points inward, and the rebuild runbook that carried the same scp is fixed. The snapshot keeps its place in the encrypted archive, because riding inside the backup is the one thing git cannot do for a host with no access to it.

What is still open, and it is not this issue

Installing the current running version at that snapshot path, so a rebuild restores the script that actually runs rather than one hand-written from a description. Tracked separately, since it is a host action rather than a change here.

Why the timer is not proof yet

It has copied, but every copy so far was started by hand. LAST is still - and the journal holds one entry, so the first genuinely scheduled run has not happened. One copy is a fact; "backed up daily" is still a unit file until that fires.