Skip to content

feat(agents): resume the incident responder - #870

Merged
mashkovd merged 1 commit into
mainfrom
feat/unsuspend-incident-responder
Aug 16, 2026
Merged

mashkovd merged 1 commit into
mainfrom
feat/unsuspend-incident-responder

Conversation

@mashkovd

Copy link
Copy Markdown
Member

Что меняется

mctl-agents-incidents: suspend: true → false. Расписание не трогается — 15 * * * *, concurrencyPolicy: Forbid.

Два блокера, и это были разные блокера

Приостановлен 29 июля. Важная деталь: причина, записанная в файле, и причина, которую все помнят, — не одно и то же. Проверил обе по факту, а не по changelog.

1. Записанная здесь: незасеянный claude-code-oauth-token-2. Без второго аккаунта fallback-шаг валил весь прогон.

vault kv get secret/platform/mctl-agents
  claude-code-oauth-token:   len=108
  claude-code-oauth-token-2: len=108   <- засеян

2. Та, которую помнят: диагностированные тикеты вечно висели в analyzing. Чинит статус escalated из mctl-agent 1.16.0.

релиз 1.16.0  — 2026-08-16T16:44:19Z
живой под     — admins-mctl-agent-base-service-78cb8597bd-94642
                ghcr.io/mctlhq/mctl-agent:1.16.0   (1/1 Running)

Снять suspend потому, что починен второй, пока первый ещё держится, — значит вернуть ровно тот отказ, ради предотвращения которого приостановка и делалась. Они разошлись случайно и закрылись тоже случайно вместе.

Отдельно: релиз выйдет как 1.16.0, а не 1.15.7 — коммит помечен feat:, release-please делает minor. Если где-то гейт записан как «после 1.15.7», это значение уже не появится.

Что остаётся приостановленным и почему

Проверил соседей — у них свои, всё ещё действующие основания, поэтому трогать их нельзя:

CronWorkflow Причина Снимать?
mctl-agents-implement Phase A: пока backlog из 12 открытых PR не получит явных исходов нет
mctl-agents-issue-poll вытеснен Temporal IssuePollWorkflow никогда

Проверка после мержа

Первый прогон — в ближайшие :15. Смотреть:

kubectl get cronworkflows -n argo-workflows   # SUSPEND=false
kubectl get workflows -n argo-workflows | grep incident

Признак, что блокер 1 действительно снят: если основной аккаунт упрётся в лимит, shepherd-fallback/run-fallback должен отработать на token-2, а не свалиться с exit 1.

Контекст

Последний пункт вечернего плана по preprod. Смежное: mctl-api 4.32.6, mctl-docs#85, gitops#867.

Suspended 2026-07-29 alongside the implementer. Both blockers are now gone,
and both were checked against Vault and the cluster rather than taken from
a changelog.

The reason written in this file was an unseeded claude-code-oauth-token-2:
with no second account to retry on, the fallback step failed the whole run
rather than masking it. Vault secret/platform/mctl-agents now carries that
field beside claude-code-oauth-token, both 108 chars.

The reason people actually cite is different — diagnosed tickets sitting in
analyzing forever — and it is fixed by the escalated status in mctl-agent
1.16.0, released today and live on admins-mctl-agent-base-service. Worth
noting the two are separate blockers that happened to clear together;
resuming because the remembered one is fixed, while the recorded one still
held, would have reintroduced exactly the failure this suspension existed
to prevent.

The sibling CronWorkflows stay suspended, on grounds that still hold:
implement until the Phase A backlog of 12 open PRs has explicit outcomes,
issue-poll permanently, superseded by the Temporal IssuePollWorkflow
schedule.

Cadence is unchanged at :15 hourly, so the first run lands within the hour
and Forbid still prevents overlap.
@cursor

cursor Bot commented Aug 16, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@claude

claude Bot commented Aug 16, 2026 •

Copy link
Copy Markdown

Claude finished @mashkovd's task in 49s —— View job


Code Review

Commit: 37bbeca9b765941ed0e5ca58bb20d2d8a1706433

Summary

  • Analyze diff and review changes
  • Submit approval verdict

Findings

This is a trivial configuration change: toggling suspend: true → false on the mctl-agents-incidents CronWorkflow plus comprehensive documentation.

No issues found — the change is straightforward and the added documentation is thorough:

  1. ✅ Both blockers verified against actual cluster state, not just changelog:

    • Vault secret claude-code-oauth-token-2 now seeded (both tokens verified at 108 chars)
    • mctl-agent 1.16.0 deployed live with escalated status fix for stuck tickets
  2. ✅ CronWorkflow spec is valid:

    • Schedule 15 * * * * and concurrencyPolicy: Forbid unchanged and intentional
    • References correct ClusterWorkflowTemplate and parameters
    • No security or operational safety concerns
  3. ✅ Sibling CronWorkflows left suspended with documented reasons

    • implement: Phase A backlog decision
    • issue-poll: superseded by Temporal workflow

Verdict: ✅ APPROVED — No P1/P2 findings. Good to merge.


@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trivial config: suspend flag resumed after both blockers verified (Vault token seeded, mctl-agent 1.16.0 deployed). Documentation comprehensive.

@mashkovd
mashkovd merged commit d96e1da into main Aug 16, 2026
10 checks passed
@mashkovd
mashkovd deleted the feat/unsuspend-incident-responder branch August 16, 2026 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant