Skip to content

Stop K3s system tests hanging on reserved pod sandboxes #6109

Description

@devantler

🤖 Generated by the Daily AI Engineer

Description

The Docker/K3s system-test lane can wedge after nested containerd leaves a pod sandbox name reserved. The cluster never becomes usable, failure diagnostics begin, and the job then remains alive until the runner receives a shutdown signal. This has now recurred independently of the code under test, including the AWS SDK-only update in #6105.

Steps to Reproduce

  1. Run the K3s + Docker + Argo CD system-test matrix lane on a GitHub-hosted runner.
  2. Observe kubelet repeatedly fail FailedCreatePodSandBox with failed to reserve sandbox name.
  3. Observe the failure path enter diagnostics/artifact handling without terminating before the workflow timeout.

Expected Behavior

The harness detects the unrecoverable nested-containerd reservation, performs bounded cleanup or one bounded recovery attempt, captures diagnostics, and exits promptly with an actionable failure.

Actual Behavior

Run 29271422238 job 86896480370 repeated the reservation error for more than 25 minutes, then remained alive until the runner shutdown roughly two hours after job start. Required Checks failed only because that lane was cancelled.

Environment

Logs

Acceptance Criteria

  • Detect the reserved-sandbox failure signature without waiting for the outer workflow timeout.
  • Bound cleanup/recovery and diagnostic upload so the job always terminates.
  • Preserve diagnostics for the original containerd/kubelet failure.
  • Add a regression test or deterministic harness check for the failure path.
  • Confirm the K3s system-test lane completes normally after the fix.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions