fix(ci): EKS smoke test cluster create times out at 30 minutes with no progress output

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

First post-#6363 dispatch of System Test - EKS (run 30201658347, 2026-07-26, main, defaults: Flux / us-east-1):

##[error]The action '🧪 ksail cluster create' has timed out after 30 minutes.

The step is capped at timeout-minutes: 30 while the job allows 120. Everything ahead of it passed — OIDC assumption, STS identity, permissions-boundary resolution, eksctl install, ksail project init. Teardown then reclaimed the cluster, so nothing leaked.

The create step emitted no output at all across those 30 minutes. The job log goes straight from the step's ##[group]Run banner at 12:21:47Z to the timeout at 12:51:59Z — 23 log lines in the whole window, all of them belonging to the teardown that followed. There is no way to tell from this run whether the control plane was still provisioning, the managed nodegroup was mid-rollout, or the command was wedged.

Problem / audience

This is the last blocker to the EKS smoke test ever completing end to end. The workflow has never had a genuine pass — the one "success" in its history (2026-07-19) skipped both the build and the smoke job, and the four real attempts before today failed on a cluster-naming defect fixed in #6308.

Two separate things are wrong, and the second is the one that will keep costing runs:

1. The budget is too small. An eksctl EKS create is control plane (~10 min) plus a managed nodegroup plus addons, and this run also had a GitOps bootstrap behind it. 30 minutes is not a generous allowance for that; it is roughly the expected duration.
2. The step is silent, so a timeout teaches us nothing. A 30-minute wall with no progress output means every failure looks identical — slow, hung, and stuck-retrying are indistinguishable, and each diagnosis costs another billable cluster.

Smallest useful change

• Raise the create step's budget to something that fits an EKS create with headroom, staying inside the 120-minute job cap and leaving room for the always() teardown.
• Make the create observable: surface eksctl/provisioner progress as it happens rather than buffering it, so the log shows which phase it reached.

Fixing only the timeout would likely turn this red run into a slower red run with the same absent diagnosis.

Acceptance criteria

• ksail cluster create for EKS completes within its step budget on a normal run, or fails with a log that names the phase it was in.
• The step streams progress while provisioning — a reader can tell control plane from nodegroup from addons.
• The teardown still runs and still verifies absence after a create timeout (unchanged from fix(ci): judge EKS smoke teardown by whether a cluster remains #6363).
• One end-to-end green run of System Test - EKS is recorded on the issue.

Notes

Cost: each diagnosis attempt creates a real, billable cluster, so batch changes rather than dispatching per tweak. Teardown was verified working on this run — ksail cluster delete itself failed (#6368) and the eksctl fallback reclaimed everything.

This is the remaining acceptance criterion for devantler-tech/platform#2327, whose credential half is now proven: OIDC assumption and sts get-caller-identity both succeeded on this run. Rough size: S–M.

Blocker: maintainer dispatch of the billable System Test - EKS workflow from main to record one end-to-end green run (budget fix #7008 and progress streaming #7081 are merged) | authority | last-verified 2026-09-17: last EKS system-test run 34999766125 (2026-09-15) predates both fixes | asked slack 2026-09-17

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• fix(ci): EKS smoke test cluster create times out before EKS can finish #6371

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[devantler]

🤖 Generated by the Agentic Engineer

Verified against the run log — this issue's reading is the correct one

Re-derived from run 30201658347's full log, because the duplicate #6371 reached the
opposite conclusion about the same run and the two diagnoses imply different fixes.

Confirmed: the create step produced no output at all. Filtering the log to the
step's whole 12:21:47Z→12:51:59Z window yields 44 lines, and the first is the timeout
itself — everything else in that window belongs to the cleanup step that follows.

#6371's contrary evidence is misattributed. It states that when the step was killed
the log still read waiting for CloudFormation stack "…-nodegroup-default". Every
waiting for CloudFormation stack line in this run is timestamped 12:52:07Z or later
— i.e. after the 12:51:59Z timeout — and each is a teardown line (will delete stack
/ waiting for stack … to get deleted). None of them is create-phase progress.

Why the distinction changes the fix

If the step were genuinely waiting on the nodegroup stack, "raise timeout-minutes"
would be the fix. It was not — it was silent, which means either ksail cluster create is buffering/suppressing eksctl's progress output, or it stalled before doing
anything observable.

So raising the cap alone would spend up to 120 minutes of billable AWS per attempt and
still produce no signal about where the time went. Progress visibility is the
prerequisite, not a nice-to-have — without it the next failure is equally undiagnosable.

Suggested order: make the step stream eksctl progress → re-dispatch → size the timeout
from the observed phase durations.

(Also confirmed while verifying: OIDC credential assumption, STS identity and
permissions-boundary resolution all succeeded — the credential path is not implicated.)

[devantler]

🤖 Generated by the Agentic Engineer

Triage note: this issue and #7007 describe the same timed-out create step, but they are not duplicates.

• Budget half: EKS smoke test: the create step's 30-minute timeout is shorter than EKS provisioning takes #7007 (opened 2026-09-12) covers only the 30-minute budget, and draft PR ci(eks): give cluster create a budget EKS provisioning can actually fit #7008 raises it to 60 minutes. That PR is waiting on one billable System Test - EKS run before it can be promoted.
• Observability half: still open, and only tracked here. Nothing in ci(eks): give cluster create a budget EKS provisioning can actually fit #7008 makes the create step stream progress, so a timeout still would not name the phase it reached (control plane, nodegroup or addons).

So #7008 should close #7007 but not this issue. When #7008 merges, this stays open for the second and fourth acceptance criteria: streamed progress and one recorded end-to-end green run.

[devantler]

🤖 Generated by the Agentic Engineer

Triage note, 2026-09-15: this issue is not a duplicate of #7007, but the two overlap.

• Budget (first bullet): covered by EKS smoke test: the create step's 30-minute timeout is shorter than EKS provisioning takes #7007 and its draft ci(eks): give cluster create a budget EKS provisioning can actually fit #7008, which raises the create step from 30 to 60 minutes. ci(eks): give cluster create a budget EKS provisioning can actually fit #7008 is parked until one billable System Test - EKS dispatch can prove it.
• Observability (second bullet): not covered anywhere. ci(eks): give cluster create a budget EKS provisioning can actually fit #7008 changes only the timeout, so a slow create still produces a silent log. That remaining scope keeps this issue open: stream the eksctl progress so the log names the phase it reached.

Once #7008 merges, this issue should be narrowed to the progress-streaming acceptance criteria.

[devantler]

🤖 Generated by the Agentic Engineer

What's left, 2026-09-17: both engineering halves are merged.

• Budget: ci(eks): give cluster create a budget EKS provisioning can actually fit #7008 gives the create step 60 minutes.
• Observability: fix(eks): show eksctl progress and the failure cause during cluster lifecycle #7081 streams eksctl's progress and output to the log, redacted, and a failure now reports the cause.

The only open criterion is one recorded end-to-end green run. System Test - EKS is a manual workflow that bills AWS, and scheduled runs don't start it. The last run (34999766125, 2026-09-15) happened before both fixes. I've asked the maintainer to dispatch it and added a blocker line above. If the run fails, its log will name the phase it reached, and that becomes the next fix.