flake: production Kind Clear/Fork lifecycle intermittently returns HTTP 409 #2401

Failure

TestKindExecutionProductionClearForkLifecycle intermittently receives HTTP 409 while publishing the first fork after the scripted source prompt completes.

• PR: test: add opt-in workspace verification and cross-module workflow #2399
• Failing commit: b3492f084a0e91b88e3811ee5ddd142dfbd81e16
• Failing execution-provider production-profile job
• Workflow: k8s-e2e; command: task e2e:k8s:execution:production
• Failure at e2e/k8s_execution/production_qualification_test.go:1386:

stage=scripted_source step=prompt_stream elapsed=1m26.528s
stage=scripted_source elapsed=1m39.526s next=source_ready
stage=source_ready elapsed=148ms next=publish_successors
production_qualification_test.go:1386: fork session status=409
--- FAIL: TestKindExecutionProductionClearForkLifecycle (101.72s)

Intermittency and PR relationship

• The same test passed in 123.23s in the previous PR-head job, at d179b0578101d0ff7dd1fbc6ac00028f19587b41. That overall job was later canceled by the subsequent push; this individual test completed successfully before cancellation.
• The production-profile job also passed on the PR's exact main base, e17d8394d365a94fb79258fa02a940625bfb99ff.
• test: add opt-in workspace verification and cross-module workflow #2399 does not change this test, server/provider/controller implementation, production fixture, production qualification task, Kubernetes workflow, or dependency pins. It changes offline workspace/module tooling, metadata policy, docs, and a separate terminal startup-test socket fixture.
• Classifying as a flake based on passing/failing runs with the relevant execution path unchanged. No same-SHA rerun or local Kind reproduction has been performed; the exact cause is not yet established.
• Searched open/closed issues for the test name, fork-status signature, successor/409, and execution-related flakes, including CI: track flaky tests on main and PRs #2327; no matching dedicated issue found.

Investigation

source_ready establishes attachment readiness, not absence of an active run/operation. successorSession (production_qualification_test.go:1469) retries HTTP 503, but reports only status on HTTP 409 and discards the response problem code. The roughly two-second interval could include a preceding 503 retry, rather than one slow fork request.

A provider successor-reservation conflict maps through ErrPlacementChanged to HTTP 409. The reservation rejects active operations, unhealthy fences, and active run claims (internal/adapter/executioncontroller/store_lifecycle.go:177–192). Kubernetes status-update retry exhaustion is another conflict path. An unsuccessful execution-run release can also leave provider state behind: server cleanup logs the release error and continues without retry (internal/adapter/server/service.go:9469–9488). Session lease competition is a distinct HTTP-409 alternative.

These are hypotheses, not confirmed causes. A simple terminal-event-before-local-drain explanation is insufficient: the prompt helper consumes the whole SSE response, normal handler cleanup releases execution ownership before returning, and the local busy-session gate normally returns HTTP 412 rather than the observed 409.

The sanitized failure artifact was inspected. It is collected after the remaining suite tests and omits the distinguishing HTTP problem code, release/reservation diagnostics, and failure-time active-run state. Its later environment snapshots cannot identify the original conflict.

Follow-up

• Preserve the sanitized HTTP problem code for every failed successor attempt, distinguishing placement conflict from session lease contention; do not log credentials or arbitrary response bodies.
• Capture bounded failure-time reservation/release reason classes, active-run/operation presence and expiry, fence health, and Kubernetes CAS retry outcome.
• Reproduce repeated Clear/Fork qualification under the production Kind profile and fix the confirmed cause with a targeted regression test.
• Do not blindly retry all HTTP 409 responses or weaken run-exclusion/ownership checks to make the test pass.

Related flake tracker: #2327.