Repository navigation
VictoriaMetrics has no backups: the vmbackup sidecar is declared but never runs #1071
Description
What
platform-gitops/bootstrap/templates/observability/monitoring.yaml:50-67 declares a
vmbackup sidecar that backs up VictoriaMetrics to Cloudflare R2 daily. It does not
exist in the running pod.
$ kubectl get pod vmsingle-... -n monitoring \
-o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'
vmsingle
The VMSingle CR has no extraContainers and no second container at all, so the
extraContainers: block sits under a values path this chart does not consume for
VMSingle. The victoria-backup-r2 secret exists and has since 2026-03-21, which is
what makes this look configured.
Net effect: the cluster's entire metrics history — every SLI the error-budget policy
reads, every alert's evaluation history — has no backup, and the manifest says
otherwise. Nothing is red, because a container that was never created cannot crash.
Two further defects in the declared sidecar, for when it is made to run
Both found by the agy reviewer on #1068 and verified here.
-storageDataPath=/storageis the wrong path. The operator mounts the VMSingle
data at/victoria-metrics-data— confirmed on the live pod.vmbackupwould find
no snapshots directory.- Errors are swallowed. The sidecar runs
while true; do /vmbackup ...; echo "Done"; sleep 86400; donewith noset -eand no exit-code check, so any failure —
wrong path, unset$S3_ENDPOINT, a network error — prints "Done", sleeps 24h and
repeats. The container never crashes, soPodCrashLoopingstays silent. This is the
same shape as the 2026-08-16 Vault-backup incident: a correct signal that reached
nobody.
Why now
retentionPeriod was raised 14d → 28d in #1068, so there is now twice as much
unbacked-up history.
Suggested fix
Make the sidecar actually render (check the chart's supported key for extra containers
on VMSingle), then -storageDataPath=/victoria-metrics-data, /bin/sh -e -c, and an
alert on backup age rather than on the container's liveness — a backup that silently
stops is the failure mode that matters, and container health does not express it.
Verification that it is fixed
kubectl get pod -n monitoring -l app.kubernetes.io/name=vmsingle \
-o jsonpath='{.items[0].spec.containers[*].name}' # must list vmbackup
kubectl logs -n monitoring <pod> -c vmbackup --tail=20
plus an object listing under s3://vault-backup/victoria-metrics with a recent
timestamp.
Found while reviewing #1068; out of that PR's diff, hence a separate issue.
Activity
mashkovd commentedon Sep 10, 2026
Confirmed from the R2 side: the bucket has never received a single object from vmbackup.
A full listing of
s3://vault-backup/today (boto3, credentials from Vaultplatform/vault/r2-backup):There is no
victoria-metrics/prefix, and zero keys anywhere in the bucket contain the stringvictoria. So this is not "the sidecar runs but fails to upload" — nothing has ever been written to-dst=s3://vault-backup/victoria-metrics, at any point since the secret was created on 2026-03-21.Two things this adds to the issue:
There is no stale backup to fall back on either. The
victoria-metricspath is not just out of date, it does not exist. Any restore attempt would fail atvmrestore -src=...with a missing prefix, not with old data.One more reason the sidecar cannot render.
platform-gitops/bootstrap/templates/observability/monitoring.yaml:593-597declaresso even once the values path is corrected, ArgoCD will not reconcile
extraContainersonto the liveVMSingle. That block has to go at the same time, or the fix will look applied and change nothing — the same silent-success shape the issue is already about.Also worth recording while the listing is fresh, because it corrects an assumption made elsewhere: the 5.84 GB in this bucket is entirely CNPG, and its retention is working correctly. 33 base-backup objects covering 2026-08-26..2026-09-10 and 1470 WAL objects over the same window — nothing outside the 14-day policy except a single 90-byte
00000005.history.gz, which barman keeps on purpose. The bucket grows ~0.29 GB/day of WAL plus a base backup that has gone from 86 MB to 111 MB over those 16 days; that is the database growing, not a leak.Found while working through the Cloudflare/R2 weekly audit.
mashkovd commentedon Sep 10, 2026
Fixed and verified in production. #1157 made the sidecar run; #1158 fixed the last defect that only surfaced once it did.
The sidecar had seven independent defects, any one of which was fatal — which is why nothing was ever red. Three were in the issue; the rest were found by running the image and watching the first real rollout:
extraContainers; VMSingle's CRD field iscontainersignoreDifferenceson/spec/extraContainerstold ArgoCD not to reconcile the very field being set — removed/vmbackup-prod, not/vmbackup(stat /vmbackup: no such file)-snapshot.createURL/-snapshot.deleteURL; the declared names do not exist and Go's flag parser exits non-zero on them-storageDataPath=/storage; the operator mounts data at/victoria-metrics-datalocalhostin the snapshot URLs. The image is Alpine, busybox resolveslocalhostto::1first, and vmsingle listens IPv4-only, so every backup died ondial tcp [::1]:8428: connection refused. Found only by watching the first rollout —127.0.0.1:8428/healthanswersOKfrom the sidecar whilelocalhost:8428is refused.No
set -e, deliberately. This is a sidecar: a container in CrashLoopBackOff is not ready → the pod is not ready → it leaves the vmsingle Service → metrics, alerting and every SLI go dark. Crashing on a failed backup trades a backup outage for a monitoring outage. Instead it publishes a success timestamp to Pushgateway and two alerts watch its age, which is what this issue asked for.Current state:
Also corrected
docs/runbooks/restore.md, which was wrong in three ways about this: it claimed 3d retention (it is 28d), called metrics loss tolerable (the rolling-28d SLI and the error-budget merge-freeze policy depend on that history since 2026-09-06), and itsvmrestoreexample passed a snapshot name under-srcwhen vmbackup keeps an incremental copy at the destination path itself.Note the R2 cost, since it is new: this writes ~5 GB today, growing toward ~8.4 GB as 28d retention fills. vmbackup deletes destination objects the current snapshot no longer references, so it is bounded by VM retention rather than open-ended — about $0.15/month on the account's overage. Accepted.
Not carried over: a restore drill. The runbook's drill journal has CNPG, Vault and etcd but never VictoriaMetrics — there was nothing to drill against until today. Worth doing once the prefix has a few days of objects; that is its own task, not this issue.