Skip to content

fix(ci): EKS smoke cleanup fails when there is nothing to clean up #6362

Description

@devantler

🤖 Generated by the Agentic Engineer

The EKS smoke test's cleanup step reports failure when there is nothing to clean up, which destroys the one signal that is supposed to mean "a billable cluster may be stranded".

Evidence

System Test - EKS run 29822971789 (2026-07-21) ended with two failed steps:

  • 🧪 ksail cluster create — the causal failure.
  • 🧹 Delete EKS smoke cluster — exited 1 on
    ResourceNotFoundException: No cluster found for name: st-eks-29822971789-1.

Creation had failed before any cluster existed, so there was genuinely nothing to delete — yet cleanup still failed the job.

Why it matters

The step sets attempted=true before invoking ksail cluster create, so cleanup always runs once creation is entered. Both delete paths then 404 and the step propagates that non-zero status:

ksail cluster delete --provider AWS --name "$KSAIL_EKS_CLUSTER_NAME" --force
delete_status=$?
if [ "$delete_status" -ne 0 ]; then
  eksctl delete cluster --name "$KSAIL_EKS_CLUSTER_NAME" --region "$AWS_REGION" --wait
  delete_status=$?
fi
if [ "$delete_status" -ne 0 ]; then
  exit "$delete_status"
fi

Three consequences:

  1. A genuine stranded-cluster alarm becomes indistinguishable from noise. Cleanup failure is the only signal that says "go check AWS for a running, billable cluster". Because it now fires on every failed create, the signal that costs real money is the one being drowned out.
  2. It masks the causal failure. A reader sees two red steps and must work out which one actually broke.
  3. It makes a green end-to-end run unreachable whenever create fails for any reason.

The step asserts the wrong thing: it trusts the delete command's exit status rather than the post-condition it actually cares about — that no cluster remains.

Expected behaviour

Cleanup succeeds when no cluster named $KSAIL_EKS_CLUSTER_NAME exists in $AWS_REGION, regardless of how the delete commands exited. It fails only when a cluster is still present after the delete attempts — the case that genuinely warrants a human looking at the AWS bill.

Acceptance criteria

  • Cleanup exits 0 when the cluster is absent, including when creation failed before creating anything.
  • Cleanup exits non-zero when a cluster is still present after both delete attempts, with the warning naming the cluster and region.
  • A failed ksail cluster create leaves exactly one red step — the causal one.

Rough size: XS.

Part of devantler-tech/platform#2327, which needs a green end-to-end System Test - EKS dispatch as its remaining acceptance criterion. The unrelated eks-default naming cause behind the same run was fixed separately by #6308.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions