[mashkovd]

Follow-up to #1157 / #1071. A seventh defect in the vmbackup sidecar, found by watching the first real rollout rather than by reading the manifest.

What happened

The sidecar started, ran, and failed:

[vmbackup] starting backup to s3://vault-backup/victoria-metrics
fatal  cannot create snapshot: Get "http://localhost:8428/snapshot/create":
       dial tcp [::1]:8428: connect: connection refused
[vmbackup] ERROR: backup failed, exit 255. Retrying in 1h; no success timestamp published...

It is not a startup race

That was my first reading and it is wrong. Measured from inside the running sidecar, against a fully healthy vmsingle in the same pod:

$ wget -q -O - http://127.0.0.1:8428/health
OK
$ wget -q -O - http://localhost:8428/health
wget: can't connect to remote host: Connection refused

$ cat /etc/hosts
127.0.0.1  localhost
::1        localhost ip6-localhost ip6-loopback

The image is Alpine; busybox resolves localhost from /etc/hosts and tries ::1 first. vmsingle listens on IPv4 only. So every backup would have failed for ever, not just the first one.

The localhost URLs came unchanged from the original 2026-03 block — which is exactly why this survived six defects' worth of review. The code had never run, so it had never had the chance to fail this way.

Changes

• Both snapshot URLs use 127.0.0.1.
• A wait loop on 127.0.0.1:8428/health before the first backup. Containers in a pod start together, so even with the address fixed the first attempt can lose the race with vmsingle, and the retry is an hour away. Waiting is deterministic; retrying is not.
• Defect (7) added to the list in the manifest comment, with the measurement.

What this says about #1157

The alerting shipped there is what made this visible within minutes instead of at the next restore attempt:

• the sidecar failed and did not crash — pod stayed 2/2 Running throughout, vmsingle never left its Service;
• it published no success timestamp, so VictoriaMetricsBackupStale would have paged at 25h had nobody been looking;
• the 1h retry (added from a review P3 on that PR) meant the failure recurred on a useful cadence instead of once a day.

That is the design working as intended on its first real fault.

Verification

helm template renders and kubeconform -strict passes (84 resources). The failing and working addresses were both measured from the live sidecar, quoted above. No rule changes, so the vm-rules suite is untouched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XfiLHHnDwXvob2smFsRcKN

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[claude]

Claude encountered an error after 0s —— View job

---

I'll analyze this and get back to you.

[claude]

Claude finished @mashkovd's task in 1m 53s —— View job

---

Reviewing PR #1158

• Gather context (diff, CLAUDE.md)
• Analyze correctness / operational safety
• Post findings / verdict

Verdict: APPROVED (0 P1/P2, 0 P3)

Reviewed git diff cb6a6ed9...cb3177a1 on platform-gitops/bootstrap/templates/observability/monitoring.yaml:

• URL fix (localhost → 127.0.0.1, both -snapshot.createURL and -snapshot.deleteURL) is correct. Matches the documented root cause: Alpine/busybox resolves localhost via ::1 first, vmsingle listens IPv4-only, so the fix is the right one for the observed dial tcp [::1]:8428: connect: connection refused.
• New pre-loop health wait (while ! wget -q -O /dev/null "http://127.0.0.1:8428/health"; do sleep 5; done) closes the startup-race window before the first backup attempt. It's unbounded, but that's safe here: the vmbackup container has no readinessProbe, so it can't gate pod/Service readiness for vmsingle, and each retry is logged for visibility — consistent with the file's existing "sidecar must never crash the pod" design philosophy already documented above it.
• Endpoint/port (127.0.0.1:8428) match the live-measured values quoted in the PR description.
• No RBAC, secrets, resource-limit, or ArgoCD spec changes — scope is limited to this shell script logic, as claimed.

Approval posted: #1158 (review)

[claude]

No P1/P2 findings (0 P3). URL fix (localhost→127.0.0.1) is correct — matches VictoriaMetrics IPv4-only listener + Alpine's ::1-first resolution described in the PR. The new pre-loop wget health wait is bounded to container start (no readinessProbe on this sidecar, so it can't block pod readiness / the vmsingle Service), uses the same 127.0.0.1:8428 endpoint verified live, and logs each retry — consistent with the file's existing no-crash/no-set-e sidecar philosophy. Good to merge.

[github-actions]

Antigravity (agy) review — informational

Model: gemini-3.1-pro-high · commit cb3177a.

Severity: P3
File: platform-gitops/bootstrap/templates/observability/monitoring.yaml, line ~131
Anchor: + while ! wget -q -O /dev/null "http://127.0.0.1:8428/health"; do
Concrete failure scenario: The script adds a loop to wait for vmsingle to become available on 127.0.0.1 before attempting the first backup, avoiding a race condition that triggers a 1-hour backoff. However, there is no test visible in this diff to ensure this retry logic correctly blocks on failure or that it targets the proper IPv4 loopback address. A future change to the script could silently break this synchronization.
Proposed fix: Add a test under tests/ to verify the sidecar script's waiting behavior when the health endpoint is unreachable and to assert the correct 127.0.0.1 address is used.