prometheus-pushgateway is scraped twice, so every pushed metric is duplicated and its alerts fire twice #1159

Activity

1. mctl-agents commented on Sep 10, 2026

   

   mctl-agentsBot

   on Sep 10, 2026

   Contributor

   More actions

   mctl-agents issue-investigator has analyzed this issue and created a proposal:

   https://github.com/mctlhq/mctl-gitops/tree/main/platform-gitops/agents-state/mctl-gitops/proposals/issue-1159-prometheus-pushgateway-is-scraped-twice/

   Status: proposed — pending human approval. Review requirements.md, design.md and tasks.md, then approve: signal this issue's DevLoopWorkflow (mctl-api POST /api/v1/agents/dev-loop/dev-loop-mctlhq-mctl-gitops-1159/approve, or python -m orchestrator.temporal.cli approve dev-loop-mctlhq-mctl-gitops-1159 --approver <your-identity>) — the workflow flips .status.yaml to accepted via the mctl-agents-approve operation and runs the Tier 2 implementer. If no DevLoopWorkflow is running for this issue (pre-Temporal proposal), run the mctl-agents-approve operation directly with service=mctl-gitops slug=issue-1159-prometheus-pushgateway-is-scraped-twice.

   The approver is recorded in .status.yaml and in the gitops commit, and the implementer refuses a proposal whose control.requires_human_approval carries no named approver — so prefer the mctl-api endpoint, which takes the approver from the authenticated caller rather than from what the caller types.
2. mashkovd

   added a commit that references this issue on Sep 10, 2026

   chore(agents): investigate #1159 2026-09-10

   d5f40f1
3. mashkovd

   closed this as completedin #1160on Sep 10, 2026
4. mashkovd commented on Sep 10, 2026

   

   mashkovd

   on Sep 10, 2026

   MemberAuthor

   More actions

   Merged in #1160 (78aa9372). ArgoCD synced and pruned the VMPodScrape. All five verification checks from tasks.md pass in-cluster.

   T1 — exactly one scrape pool, up

   pools: 1
     serviceScrape/monitoring/prometheus-pushgateway/0  up  http://10.42.4.230:9091/metrics
   
   $ kubectl -n monitoring get vmpodscrape prometheus-pushgateway
   Error from server (NotFound): ... "prometheus-pushgateway" not found
   

   The podScrape/... pool is gone, and the object was pruned by ArgoCD rather than left orphaned.

   T2 — duplication gone (measured after ~7 minutes, past VM's staleness horizon, so the abandoned endpoint="9091" series had aged out)

   count by (__name__) ({__name__=~"mctl_telegram_canary_success|vmbackup_last_success_timestamp_seconds"})
   
     mctl_telegram_canary_success             -> 1   (was 2)
     vmbackup_last_success_timestamp_seconds  -> 1   (was 2)
   

   T3 — honorLabels survived

   push_time_seconds                        job=mctl_telegram_canary  endpoint=http
   push_time_seconds                        job=vmbackup              endpoint=http
   vmbackup_last_success_timestamp_seconds  job=vmbackup              endpoint=http
   

   The pushed job labels are intact, so MctlTelegramCanaryStale and the backup rules still have series to evaluate. Only endpoint=http remains.

   T4 — no alert regression

   MctlTelegramCanaryAbsent           inactive/ok
   MctlTelegramCanaryFailing          inactive/ok
   MctlTelegramCanaryStale            inactive/ok
   ScrapePoolHasNoTargets             inactive/ok
   VictoriaMetricsBackupMetricAbsent  inactive/ok
   VictoriaMetricsBackupStale         inactive/ok
   

   ScrapePoolHasNoTargets staying quiet is the one that matters — it is the safety net for "the one remaining scrape selected nothing".

   T5 — CI green on the PR, including the kubeconform sweep and the alerting-rule step.

   T6 — the guard was proved to fire. Ran locally by dropping a copy of the deleted VMPodScrape back into vm-rules/:

   exit=1
   ::error file=...zz-temp-podscrape.yaml::non-VMRule object (kind=VMPodScrape) found under vm-rules/; move it to infra-components/observability/<component>/ instead
   

   That manual check has since been made permanent. agy raised a P3 on the PR — the new guard had no automated test, so a later refactor could quietly disarm it, which is the shape of this bug one level up. scripts/check-vm-rules.sh now takes --selftest and CI runs it before the real pass, matching the convention every other detector in validate-manifests.yml already follows. It re-runs the script against a throwaway RULES_DIR rather than reimplementing the check, so it exercises the guard and not a copy of it, and it was itself validated by mutation:

   guard reverted to `skip (kind=...)` + continue  -> self-test exits 1
   annotation text changed                          -> self-test exits 1
   restored                                         -> self-test exits 0
   

   CI output on the merged commit: check-vm-rules.sh self-test: kind guard rejects non-VMRule, accepts VMRule.

   Closing.
5. mashkovd

   added a commit that references this issue on Sep 11, 2026

   fix(ci): check where the CA is fetched from, and that no template ove…

   ...

   a77bc9e