Repository navigation
feat(skill): argocd_sync_failed — diagnose stuck-degraded ArgoCD apps - #14
Conversation
ArgoCDDriftSkill only handles benign OutOfSync+Healthy drift. Apps that
are Degraded with a sync failure had no diagnostic skill and silently
piled up TypeArgoCDDegraded tickets. Most painful instance: external-
secrets stuck Degraded for 6 days (2026-05-01 → 2026-05-07) while the
agent system never proposed any recovery — see plan
~/.claude/plans/streamed-soaring-fox.md.
The new skill matches OutOfSync+Degraded (priority 60), Degraded only
(priority 55), and bumps to priority 80 + HIGH confidence when the
status message contains a known recovery-drill signature:
- "must remain in spec.versions" / "missing from spec.versions" →
CRD storedVersion conflict (chart major-version revert long-tail)
- "request to convert CR from an invalid group/version" →
managedFields poisoning from a temp CRD recovery window
Diagnose() returns the exact recovery commands as plain text. Fix() is
intentionally nil — recovery is cluster-side kubectl, not a gitops PR,
so the diagnosis lands in Telegram for human approval.
Also wires AlertManager → ticket classification: the new
ArgoCDApplicationDegraded / ArgoCDApplicationSyncFailed alerts (added
in mctl-gitops PR #142) now map to TypeArgoCDDegraded in classifyAlert
so the new skill picks them up.
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 74ee766d7d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case "ArgoCDApplicationDegraded", "ArgoCDApplicationSyncFailed": | ||
| return ticket.TypeArgoCDDegraded, ticket.SeverityWarning |
There was a problem hiding this comment.
Mapping ArgoCDApplicationDegraded/ArgoCDApplicationSyncFailed to TypeArgoCDDegraded without also extracting the app name from alert labels causes these tickets to keep service from pod (or empty), even though processAlert currently only special-cases name for workflows (internal/monitor/alerthandler.go lines 88-97). In environments where these ArgoCD alerts do not carry an application pod label (or carry only ArgoCD controller pods), unrelated app incidents collapse into one dedupe key and collectEvidence skips argocd_status when service=="" (internal/pipeline/evidence.go lines 43-50), so the new argocd_sync_failed skill cannot diagnose the actual failing application.
Useful? React with 👍 / 👎.
Codex P1 on PR #14: ArgoCDApplicationDegraded / ArgoCDApplicationSyncFailed alerts come from argocd-metrics with the Application identity in `name` (and tenant in `dest_namespace`), not in `pod`. processAlert was previously only special-casing `name` for TypeWorkflowFailed, so every Degraded application collapsed onto the same (tenant="", service="") dedup key. pipeline.collectEvidence then skipped argocd_status entirely (gated on service != ""), so the new argocd_sync_failed skill never saw the data it needed to diagnose. This adds the equivalent special-case for TypeArgoCDDegraded plus a table-driven test that asserts service=name and tenant=dest_namespace. classifyAlert tests grow two new cases for the alert names themselves.
|
@codex review |
The companion mctl-gitops PR #142 renamed the alert from ArgoCDApplicationSyncFailed to ArgoCDApplicationOutOfSyncLong after Codex P2 (the alert fires on prolonged OutOfSync drift, not a sync-failure signal). Add the new name to classifyAlert so the two PRs stay merge-order-independent. The legacy name stays in the switch until both PRs have landed and the chart has rolled out.
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
Why
`ArgoCDDriftSkill` only handles benign `OutOfSync+Healthy` drift. Apps stuck Degraded with a sync failure had no diagnostic skill — most painful case: `external-secrets` Application stuck Degraded for 6 days (2026-05-01 → 2026-05-07) with no proposed recovery, GitOps loop wedged across 11 child apps. See plan `~/.claude/plans/streamed-soaring-fox.md` and recovery drill `reference_eso_storedversion_recovery.md`.
Test plan