Skip to content

feat(workflows): add mctl-agents-shepherd CWFT + CronWorkflow (suspended) - #88

Merged
mashkovd merged 1 commit into
mainfrom
feat/cwft-mctl-agents-shepherd
May 1, 2026
Merged

mashkovd merged 1 commit into
mainfrom
feat/cwft-mctl-agents-shepherd

Conversation

@mashkovd

@mashkovd mashkovd commented May 1, 2026

Copy link
Copy Markdown
Member

Summary

Companion gitops PR for the Tier 3 PR shepherd. Lands two new files
under platform-gitops/argo-workflows/cluster-templates/ so that
once ghcr.io/mctlhq/mctl-agents:1.9.0 ships, ArgoCD can sync the
template and the operator can drive the first one-shot manually
before un-suspending the cron.

  • cwft-mctl-agents-shepherd.yaml — ClusterWorkflowTemplate
    mctl-agents-shepherd. Mirrors cwft-mctl-agents-implement.yaml
    (SSH clone-gitops, run container, .status.yaml-only commit-and-push,
    telegram onExit, shared mctl-gitops-main-writes mutex, 8Gi PVC,
    7200s deadline). Runs python -m orchestrator.run_shepherd with
    --service, --slug, --dry-run wired through env-var argv-build
    (no inline workflow.parameters interpolation in shell bodies, per
    the Argo template glob-scan footgun documented in project memory).
  • cronworkflow-mctl-agents-shepherd.yaml — CronWorkflow
    mctl-agents-shepherd. Schedule */30 * * * *, concurrencyPolicy: Forbid,
    suspend: true. Operator un-suspends after the first one-shot
    succeeds.

The image tag 1.9.0 references the upcoming mctl-agents build
that ships orchestrator/run_shepherd (mctlhq/mctl-agents#12). The
run-shepherd step will ImagePullBackOff until that tag is
published — intentional. ArgoCD picks up the new template the moment
the tag drops; no second gitops PR is needed.

Tier 3 spec lives in platform-gitops/agents-state/mctl-agents/proposals/tier3-pr-shepherd/
(requirements, design, tasks). Task 6 in that spec is exactly this
PR's scope.

Test plan

  • yq eval '.kind' … confirms both files parse to the expected
    ClusterWorkflowTemplate / CronWorkflow kinds.
  • kubectl --dry-run=client apply -f … against the live mctl-preprod
    API server accepts both manifests (CRDs validate).
  • After merge: ArgoCD argo-workflows-config Application syncs
    cleanly; kubectl -n argo-workflows get clusterworkflowtemplate mctl-agents-shepherd returns the resource.
  • Once mctl-agents:1.9.0 ships: operator submits a one-shot
    Workflow with --dry-run=true against a single proposal to verify
    the decide() path before un-suspending the cron.
  • Post-un-suspend: confirm kubectl -n argo-workflows get cron mctl-agents-shepherd -o jsonpath='{.status.lastScheduledTime}'
    advances every 30 min and concurrencyPolicy: Forbid drops
    overlaps cleanly.

…ded)

Adds the Tier 3 PR shepherd companion to mctl-gitops:

- ClusterWorkflowTemplate `mctl-agents-shepherd` mirrors
  cwft-mctl-agents-implement.yaml (SSH clone-gitops, run-shepherd
  container, .status.yaml-only commit-and-push, telegram onExit,
  shared `mctl-gitops-main-writes` mutex, 8Gi PVC, 7200s deadline).
  Runs `python -m orchestrator.run_shepherd` in
  ghcr.io/mctlhq/mctl-agents:1.9.0 with --service / --slug /
  --dry-run wired through env-var argv-build (no inline
  workflow.parameters interpolation in shell bodies, per the Argo
  template glob-scan footgun documented in project memory).

- CronWorkflow `mctl-agents-shepherd` runs every 30 minutes with
  concurrencyPolicy: Forbid. Suspended at install time so the
  operator can run the first shepherd one-shot before letting the
  cron loose.

The 1.9.0 image tag references mctl-agents PR #12 which has not
shipped yet — the run-shepherd step will ImagePullBackOff until
then. Intentional: ArgoCD syncs the CWFT the moment the tag drops,
no second gitops PR needed.

Spec: platform-gitops/agents-state/mctl-agents/proposals/tier3-pr-shepherd/
@mashkovd

mashkovd commented May 1, 2026

Copy link
Copy Markdown
Member Author

@codex review

@mashkovd

mashkovd commented May 1, 2026

Copy link
Copy Markdown
Member Author

@copilot review

Copilot AI commented May 1, 2026

Copy link
Copy Markdown

@copilot review

Here's my review of the two new files:

cwft-mctl-agents-shepherd.yaml

✅ Correctly mirrors cwft-mctl-agents-implement.yaml structure throughout (clone → run → commit entrypoint, same mutex, same PVC/deadline sizing, same continueOn: failed/error: true on the main step, same 5-retry rebase/push loop).

✅ Security: workflow params are injected via env vars (WORKFLOW_SERVICE, WORKFLOW_SLUG, WORKFLOW_DRY_RUN) and consumed with set -- argv construction — no direct {{workflow.parameters.*}} interpolation in shell bodies. Consistent with established project practice.

✅ commit-and-push correctly guards against non-.status.yaml drift before staging (the NON_STATUS_CHANGES check) and short-circuits cleanly on ticks with nothing to commit.

✅ GITHUB_TOKEN marked optional: true — intentional and consistent with the implement template (dry-run discovery path still works without it).

⚠️ Minor — SHEPHERD_BUDGET_USD: "5.00" is only effective if run_shepherd.py reads and enforces it. The implement template uses IMPLEMENTER_BUDGET_USD: "3.00" under the same expectation. If the application code doesn't wire this up it silently has no effect — worth confirming in the mctl-agents#12 PR.

⚠️ Minor — imagePullPolicy: IfNotPresent with a not-yet-published tag (1.9.0). On first scheduling the pod will always attempt a pull (image absent from node cache), so it will ImagePullBackOff until 1.9.0 ships — this is documented as intentional. No change needed, just confirming the intent matches.

cronworkflow-mctl-agents-shepherd.yaml

✅ suspend: true, concurrencyPolicy: Forbid, startingDeadlineSeconds: 600, history limits, and clusterScope: true reference are all correct and consistent with cwft-mctl-agents-daily.yaml.

Overall: no blocking issues. The two files are correct, secure, and consistent with the codebase patterns. Ready to merge once mctl-agents#12 ships 1.9.0.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mashkovd
mashkovd merged commit c4e24be into main May 1, 2026
@mashkovd
mashkovd deleted the feat/cwft-mctl-agents-shepherd branch May 1, 2026 09:31
mashkovd pushed a commit that referenced this pull request May 21, 2026
- Flip issue-86 (production alert PrometheusRule) to status=accepted.
- #86: correct namespace (monitoring) + labels (prometheus: kube-prometheus /
  role: alert-rules) to match deploy/alerts/canary.rules.yaml; drop the three
  alerts already covered by the deployed mctl-telegram-alerts VMRule
  (RateLimitWave/ClientErrorsSpike/AuthFailuresSpike); document VM-operator
  auto-conversion and the gitops mirror path.
- #88: note CRD infra already exists (canary precedent); burn-rate groups APPEND
  to #86's rules file; sequence after #86.
- #92: link canary section to existing docs/runbooks/canary.md instead of
  duplicating; map auth/client/ratelimit runbook anchors to the #59 VMRule
  alerts; fix anchor table.
mashkovd pushed a commit that referenced this pull request May 21, 2026
Accept #88 (Beta SLOs + burn-rate alerts) for implementer trigger. #86 has
merged (PR #112) and shipped placeholder runbook_url -> docs/hpa.md#alerts, so
#92 task 2 now says repoint (not just confirm) to docs/runbook.md anchors.
mashkovd pushed a commit that referenced this pull request May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants