Skip to content

feat: draft native Kubernetes execution provider - #1614

Merged
JAORMX merged 28 commits into
mainfrom
impl/native-kubernetes-execution
Sep 26, 2026
Merged

JAORMX merged 28 commits into
mainfrom
impl/native-kubernetes-execution

Conversation

@JAORMX

@JAORMX JAORMX commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Stage: qualified implementation candidate (draft, existing stack)

Plan: #1579 (plan/native-kubernetes-execution), docs/acceptance/native-kubernetes-execution.md, draft ADR 0351. This updates the existing stack only: no new PRs, no automatic merge, no implied human approval or shipment.

Historical native live qualification: PASS

Manual run 35700303737 checked out exact pre-main-integration runtime commit 2020e59934369e8f771059d0c173fa4cbf964c7e and passed the complete sequence:

  1. Deterministic production Kind+Calico qualification.
  2. Protected runtime credential staging and native real-provider configuration.
  3. OpenRouter / anthropic/claude-haiku-4.5 coding: 30,521 input / 713 output tokens, Write and Shell observed, exact model go test ./... exit 0.
  4. Independent read-only published-binding lookup, exact-reference typed gRPC Attach/run acquisition, verification of two generated files, and independent go test ./... exit 0.
  5. Mock restoration and UID-pinned Secret cleanup, private credential-file removal, exact owned-cluster deletion and absence check.

This is native execution-provider evidence for that tested runtime, not current-head qualification and not the separate mecak8s live suite. No credential bytes entered agent/tool context.

Additional direct-client validation of the local repairs: PASS

Before main integration, a direct client exercised the repaired working tree based on 90320a1b, using an operator-provided OpenRouter credential through the trusted runtime loader. This is separate from the historical CI run above; the detailed status comment records the image digests and scope.

  • Real anthropic/claude-haiku-4.5 usage: 30,525 input / 745 output tokens; Write and Shell observed; the model's exact go test ./... exited 0.
  • An independent typed client verified the generated artifacts, added fixed semantic checks, and ran tests successfully.
  • After restarting both the execution provider and mecak8s, fresh clients reloaded the same session and exact environment. Artifacts matched byte-for-byte, and independent tests again exited 0.
  • Mock configuration was restored and the run-scoped credential Secret was removed using its UID-pinned receipt. No credential contents were exposed to the agent or posted.

This validates the pre-main-integration repaired runtime, not the subsequently merged current head. Its sanitized receipt is local, not an uploaded CI artifact.

Blockers repaired

The main-integrated candidate preserves the previously reviewed repairs and adds regression coverage for:

  • shared initial-reservation retention with exclusive successor abort;
  • exact-ID retry and given-binding cleanup;
  • authority plus operation-TTL renewal/completion, including expired-idle terminal proof without force takeover;
  • publication of a matching durable placement-collision winner while retaining the pending reference on commit failure.

It also retains the earlier scoped administration, provisioning recovery, Helm confinement/history, migration/finalizer safety, immutable rotation publication, remote filesystem/CAS confinement, permission preservation, and bounded diagnostic repairs.

Current integration and local verification

Current implementation head: 4b667b833788b5918ab66b110a1fababdd569d3c. Its second parent is the exact current plan head 4ffb2c288796d515d3fd2a5ffc744382f1d45418, which includes origin/main through ca993a6ea1b648e8e18a102b6810da5dea8c494f. Mainline ADR 0350 is retained and the native draft is ADR 0351.

After the latest integration, local verification passed:

  • affected -race packages: execution controller/client, server, app, and execution Kind fixture;
  • task api:check;
  • task docs;
  • full task lint;
  • full task test (race plus standalone module hygiene).

Earlier main-integration verification also passed task build, task site:build, task ac-trace-strict, and the offline mecademo smoke test. Those commands were not rerun locally after the final two main commits. Post-integration CI results are recorded below; the additional local direct-client receipt remains pre-integration evidence.

After rerunning the unsuccessful jobs on the unchanged current head, GitHub checks are terminal: 29 successful, 8 skipped, and 1 failed, with the separate Vercel status successful. CI is not fully green.

  • Native production Kind qualification failed again on attempt 2, at the same TestKindExecutionExpiredRunAllowsFinalReferenceRetireAndDelete retained-deletion request (pr1614_runtime_blockers_test.go:142, internal: execution provider request failed). The first attempt's sanitized artifact shows retirement and executor termination succeeded, but deletion was not durably admitted. The underlying backend/transport cause remains unidentified. Focused retirement/deletion race tests passed 20 iterations; no speculative code change or weakened assertion was made. Two CI attempts now reproduce the same failure, so it cannot be dismissed as a one-off flake.
  • Generic live Kubernetes qualification passed on attempt 2. Its first attempt was cancelled by repository-wide concurrency; that cancellation is now superseded by a successful rerun. This is the generic mecak8s live suite, not the separate manual native-execution live qualification.

The historical and direct-client native live receipts above remain scoped to their tested pre-main-integration runtimes; they do not replace the remaining failed current-head check.

Both PRs stay draft. Human API/schema/security approval, release-maturity selection, and final merge review remain outstanding. No production-ready/merged/shipped claim is made.

@JAORMX
JAORMX added this pull request to stack #1618 September 16, 2026 07:51
@JAORMX
JAORMX force-pushed the impl/native-kubernetes-execution branch from 6aa84d1 to 7c9c523 Compare September 21, 2026 07:35
@JAORMX JAORMX added the e2e-live Run the live e2e suite on this PR label Sep 21, 2026
zozo123 added a commit to zozo123/mecatl that referenced this pull request Sep 23, 2026
…nce tables

Answers the review ask for a builder composition that makes the adapter
testable end to end, and lines it up with stacklok#1614.

Composition (internal/adapter/boatenv/composition_test.go): drive scripted
sessions through the real app.Build with the Boat provider as the
deployment PlacementProvider. Doing so surfaced three gaps that library
tests could not see, all fixed here:

- Startup preflight provisioned a sandbox. configuredPlacementBinder
  validated the default placement by binding it, which for Boat creates
  and archives a billable sandbox on every daemon start. Add the
  side-effect-free server.PlacementValidator seam, with the same name,
  signature and call site as stacklok#1614 so the two merge cleanly, and implement
  it for Boat as GET /me. Providers without the seam keep the legacy
  bind-to-validate path (both pinned in placement_validator_test.go).
- File tools were denied by authority: the workspace could not derive an
  authority resource identity. Implement tool.AuthorityResourceResolver
  with a guest round trip that resolves symlinks under the same
  confinement as file access, so policy and access name the same file.
- Shell needs a local workspace and shell on main even when the
  placement supplies the runner. stacklok#1614's RemoteExecution posture removes
  that gate; until it lands, the Shell composition test enables it with a
  throwaway host workspace and proves nothing runs there.

Conformance: run engine/adapter/fsconformance Run and RunNamespace
against the Boat workspace, offline over the fake API and live over one
sandbox with a directory per case. It caught a contract bug: ReplaceFile
with a zero FileVersion on a missing path reported a version mismatch
instead of fs.ErrNotExist. A zero version still goes to the guest, which
now reports a missing file before a stale version.

Live, against boat.dev: TestLiveBoat (lifecycle, CAS, archive and exact
reattach), TestLiveBoatConformance (30/30) and TestLiveBoatComposition
(app.Build, one sandbox, Write/Edit/Grep/Shell/Read) all pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JAORMX

JAORMX commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Repair and live-validation status

The three runtime blockers were reproduced and repaired locally, and a direct OpenRouter client journey passed. The repairs are still uncommitted and have NOT been pushed to this PR. The PR remains at 90320a1ba0a8ed538c888879cd4fd838467c78b4; the results below describe the repaired working tree based on that commit, not the unchanged PR head.

Reproduced failures and local repairs

All three original failures were demonstrated in both focused offline regressions and real Kind runs before fixing them:

  1. Execution continued after operation-authority loss. Renewal and completion now validate current authority and operation-lease expiry; loss cancels execution while retaining fencing uncertainty until termination is proven.
  2. An expired run claim could strand retirement after the final reference disappeared. Exact, expired, inactive claims can now be cleared atomically during lifecycle admission, including the terminal-proof path; live, malformed, mismatched claims and active operations remain refused.
  3. Explicit session-ID create retries conflicted with the existing allocation. Retries now use exact reattachment, and concurrent initial creation shares a stable reservation identity. Additional regressions cover conflicting concurrent requests, factory failures, and preventing a losing request from aborting the winner. Retain-on-close applies only to ambiguous shared initial reservations; exclusive successor cleanup still aborts its own unpublished reservation.

Local verification passed full task lint, full task test, affected-package race tests, and the offline demo. A separate full production Kind qualification of the repaired runtime passed 15 tests, with the explicitly disabled live-LLM test skipped. Later fresh-fixture attempts encountered build-timeout/tooling issues; those failures remain recorded rather than being represented as successful suite runs.

Actual OpenRouter usage: direct client, not a scripted-model suite

On the repaired runtime, a client created an authenticated session using OpenRouter / anthropic/claude-haiku-4.5 and requested a dependency-free Go module, implementation, and table-driven tests.

  • Real model usage: 30,525 input / 745 output tokens.
  • Model tools observed: Write and Shell.
  • The model's exact go test ./... completed with exit 0, and the run ended with end_turn.
  • An independent typed execution client resolved the published binding, attached to the exact environment, inspected the generated artifacts, added fixed semantic checks, and ran go test ./... with exit 0.
  • Both the execution provider and mecak8s were restarted. Fresh clients reloaded the same session and exact environment reference; the artifacts matched byte-for-byte, and independent tests again returned exit 0.

The operator-provided credential was consumed only by the trusted runtime loader, not read into agent context. The mock profile was restored afterward, and UID-pinned deletion of the run-scoped credential Secret succeeded. No credential bytes or credential-file path are included in this comment.

Tested runtime image digests:

Component Digest
Execution provider sha256:b7355c80f2238247ff710557a33273fcf6f5646dcdbeae84e328d094ec94ae9b
mecak8s sha256:cd01399bd05d203079d187f9872173b73dac8dc80a1ef17d5e9155022f6c45f6
Execution workload sha256:a5522c54eea716832e7c653e6492f38e78cd79079995fd11e607ceda4cd14b58

The sanitized local receipt records exact_ref_match: true, restart_verified: true, all three command exit codes as zero, and artifact SHA-256 hashes. It has not been uploaded as a CI artifact. This demonstrates one real coding/persistence/restart journey; it is not a claim of exhaustive workload or failure-mode coverage.

Current PR state and remaining merge work

  • Still open and draft, targeting plan/native-kubernetes-execution (docs: plan optional native Kubernetes execution provider #1579), not main.
  • Current PR-head checks: 30 successful, 8 skipped; no failures or pending checks. These checks do not validate the unpushed repairs.
  • GitHub reports the current PR/base relationship as clean.
  • A non-mutating merge simulation against current main (b0cf03a423df907e07269e085105fda2dc81ea47) finds conflicts in:
    • .claude/skills/to-acceptance-plan/scripts/check-acceptance-plan.sh
    • docs/acceptance/README.md
    • docs/adr/0027-cloud-native.md
    • internal/adapter/server/classification.go
    • internal/adapter/server/service.go
    • internal/app/build.go

Remaining: record the repairs/evidence in the reviewed stack, commit and push the fixes, reconcile current-main conflicts, and rerun checks on the resulting candidate. Human API/schema/security approval, plan merge, release-maturity selection, and final implementation merge remain separate gates. No approval, merge, or shipment is implied by this validation.

@JAORMX

JAORMX commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Current-head qualification: PASS

The reviewed lifecycle and Helm repairs are committed and pushed as b5410380d8a1a9b20dbdb64f509a1a7139d6beba. The implementation worktree is clean.

Fresh production qualification

Deleted the failed, previously consumed qualification cluster and created a new owned Kind+Calico fixture. The full native production suite passed on this commit in 1,255.470 seconds, including the retained-environment deletion regression and the Helm lifecycle coverage. This is fresh-fixture evidence, not a rerun against already-consumed migration/authority history.

The repairs cover terminal retirement, rejection of new replacements after retirement while preserving exact receipts, profile-independent admitted retirement/deletion, fail-closed profile validation before every replacement phase, bounded backend diagnostics, and Helm 4 client-side lifecycle reconciliation. The historical generic Internal response's precise backend cause remains unidentified; the passing regression is not a claim that its raw cause was recovered.

Real-provider user journey

PASS using OpenRouter / anthropic/claude-haiku-4.5:

  • 30,627 input tokens and 739 output tokens.
  • Model-driven Write and Shell; the model's exact go test ./... exited 0.
  • An independent typed-gRPC client verified both generated artifacts and independently ran tests successfully.
  • Both the execution provider and mecak8s were restarted and awaited.
  • A fresh authenticated client reattached the same exact environment reference, verified byte-identical artifacts, and reran tests with exit 0.

This exercised authenticated HTTP session creation/model execution plus independent gRPC verification and restart recovery. It was not a manual TUI/browser walkthrough. The sanitized local live-summary.json receipt is retained; it has not been uploaded as a CI artifact.

Verification and cleanup

  • Local build, tests, race tests, lint, docs, site build, and offline demo passed.
  • CI run 35955887081 tested this exact commit: 30 successful jobs, 1 skipped. It was manually dispatched because this push did not instantiate Actions workflows. Native production/live evidence above is local qualification, not a claim that the separate native/perf workflows ran on GitHub.
  • Mock mode was restored, both deployments are ready, the run-scoped provider Secret was removed using its UID-pinned receipt, and the operator's credential source was left untouched.
  • The new healthy qualification cluster is retained for inspection.

The PR remains draft; these results do not substitute for the outstanding human contract/security/release-maturity and merge decisions.

@JAORMX

JAORMX commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor Author

Implementation handoff: native Kubernetes harness context

Align PR #1614 with the native plan amendment in #1579 and the shared harness-context contract/implementation in #1878 and #1875. This is the runtime assignment; it does not replace the native plan's human approval gate.

Before starting

Check the current merge, approval, and CI state of all four PRs. Start runtime integration only after the amended #1579 and normal shared-plan/shared-implementation merge gates, or a separate explicit directing-human exception for this integration. Original shared plan #1814 is merged; the pre-merge exception for #1875 does not authorize native runtime work. This comment grants no merge, force-push, paid/live-test, or destructive-state authority.

Use the merged shared implementation or current explicitly authorized stack, including default-child instruction inheritance and source-lifecycle fixes. Historical handoff hashes are not dependency pins. Record the candidate actually tested in completion evidence.

Read AGENTS.md, docs/development-process.md, the approved native plan/owning ADR, and the shared docs/acceptance/harness-context.md / ADR 0359. The shared contract is the narrow 25-AC, six-scenario version. Inventories, recovery recipes, history transfer, SessionAccess, and new storage fencing are outside this work.

Source model and native scope

Independent deployment/operator/API instruction and command sources work without attaching execution. Repository files become context only when separately selected and admitted. Leave PVC-backed context unselected in the native first slice. Do not implement a PVC source adapter or expand native delegation/schedule support under this assignment.

For registrations that actually require execution files, the shared internal contract is UsesExecutionWorkspace plus principal-scoped SessionID/owner/profile and selector-free AcquireExecutionWorkspace (ExecutionWorkspaceAcquirer, shared WithExecutionWorkspace methods). It yields an exact authorized read-only borrow, with no runner or execution ledger evidence, and release owns only that borrow. No public RPC field, new SessionLease, root inference, or authority grant is introduced. Independent native sources do not need this acquisition capability.

Implementation work

Inspect current internal/app/project_ingestion.go, internal/app/build.go, internal/app/remote_execution_test.go, cmd/mecak8s/main.go, and internal/adapter/executionclient/client.go. Locate blanket suppression in project admission, command listing, and applyRemoteExecutionPosture/NoopExpander after integrating the shared changes; do not assume historical line numbers or duplicate code already removed upstream.

  1. Register configured independent deployment/operator/API instruction and command sources at production composition before app.Build.
  2. Remove only execution-kind coupling that suppresses otherwise admitted context. Retain trust, required-source failure semantics, permission roots, configured Ask/Deny, credentials, hooks, and tool scope. Do not bulk-delete RemoteExecution checks.
  3. Route assembly, command discovery, and slash expansion through the shared source policy. Preserve existing root-file precedence where file-backed independent sources are used, per-run instruction refresh, and live commands.
  4. Keep independent discovery free of native provider/PVC attachment, including provider failure and no-FS execution. Unselected PVC files must neither be read nor acquired.
  5. Preserve exact EnvironmentRef placement, controller/provider/PVC/runner fencing, background-Shell rejection, and all approved native exclusions for delegation, schedules, and worktree operations.
  6. Rebind current policy and authorization after restart without deriving sources from stored placement, backend kind, or workspace paths. Preserve the shared cancellation and cleanup fixes: no late cancelled binding publication, immediate attempt cleanup/retry, and process-source retention through in-flight creation and attempt cleanup.

Required proofs

Use production Build and the public service/discovery path through the actual execution-client composition boundary:

  • Positive deployment/API markers reach model instructions, listing, and expansion. Conflicting PVC markers remain absent and acquisition count stays zero.
  • Independent discovery succeeds while the native provider is unavailable, without attach/reattach calls.
  • No-FS retains independent instructions/commands while file tools and runner remain absent.
  • Required independent-source failure is explicit, not empty success or PVC/host/global fallback.
  • Two Builds observe current policy and revoked authorization without stored-placement inference.
  • Source reads leave execution ledgers unchanged and cannot authorize Edit.
  • Exact placement, Ask/Deny, provider fencing, and existing unsupported-operation behavior remain effective.
  • Failed/cancelled binding and shutdown preserve attempt ownership and do not reintroduce the shared lifecycle bugs. Keep upstream regression tests intact; do not add unsupported native child capabilities merely to exercise shared child tests.

Reference-only tests do not qualify the actual native composition boundary. Preserve the native PR's existing production/live qualification requirements and distinguish historical results from evidence on the integrated candidate. This context assignment does not establish that unrelated native runtime issues are resolved.

Verification and completion

Workers run named proofs, affected app/server/execution-client tests and direct boundaries, scoped lint, and targeted race tests. The integration owner runs aggregate task test, task lint, and task test:race on the assembled candidate, plus the offline demo and applicable docs/site/API/generation/trace checks. Follow current verification ownership; do not run the aggregate suite in each worker. Build through the Taskfile. Paid/live tests require separate authorization.

Report the registration and suppression changes, preserved exclusions, actual tested revision, command exit codes, current CI, and real-backend versus reference evidence. Check approval state again before claiming completion. Stop for material contract drift, ambiguous source authority, or a newly required PVC adapter. Humans approve and merge.

@JAORMX
JAORMX force-pushed the impl/native-kubernetes-execution branch from b541038 to 3d5b794 Compare September 25, 2026 11:16
Base automatically changed from plan/native-kubernetes-execution to main September 25, 2026 21:00
@JAORMX
JAORMX force-pushed the impl/native-kubernetes-execution branch from 7ba64d3 to 77f456d Compare September 25, 2026 23:26
Preserve the integrated implementation tree at the last main merge while leaving the merged plan files authoritative.

Co-Authored-By: OpenAI Codex <codex@openai.com>
JAORMX and others added 20 commits September 26, 2026 09:01
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Resolve the upstream reservation and harness-instruction composition changes without restoring superseded native integration code.

Co-Authored-By: OpenAI Codex <codex@openai.com>
Integrate native awaiting-handoff preservation with upstream Studio continuation persistence. Freeze the projection decision under persistMu, then release it before EventLog I/O; terminal admission and matching retractions prevent preserving an already-settling handoff even if the append outcome is uncertain.

Restore the lost teamService provider-interface test delta. Keep FinishRun identity checks under the registry lock so stale plan relays cannot race successor promotion. Add deterministic blocked-append, persistence, drain, and ambiguous-error regressions.

Co-Authored-By: OpenAI Codex <codex@openai.com>
…tion

Join native renewal and release the drained plan run before removing its operation pin or admitting the reserved continuation. Keep the per-session entry barrier and exact identity recheck while provider I/O runs outside the global mutex.

Apply the existing native acquisition and renewal seams to restored exact plan admission too. Promote the verifier diagnostic into deterministic offline regressions for claim ordering, exclusive continuation, failure/cancellation, and stale relay cleanup.

Co-Authored-By: OpenAI Codex <codex@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Adapt native lifecycle and TUI recovery tests to the merged placement ownership, governance-root, and typed-scrollback contracts while retaining side-effect-free native preflight.

Co-Authored-By: OpenAI Codex <codex@openai.com>
Publish a competing winner only after the losing placement has been provisioned. Prove exact candidate rollback and winner lifetime in both shared and per-session engine paths, and separately prove known explicit-ID retry performs no allocation or rollback.

Co-Authored-By: OpenAI Codex <codex@openai.com>
Require all release publishers to traverse the immutable-ref validator, expose guard outputs only through direct dependencies, and make manual dispatch select the requested tag ref.

Co-Authored-By: OpenAI Codex <codex@openai.com>
Validate event SHA agreement for dispatch as well as push and cross-check the guard output against the immutable validator checkout. Cover moved tags between on-main commits with real offline fixtures.

Co-Authored-By: OpenAI Codex <codex@openai.com>
Preserve the pre-existing commit co-author ADR as 0367 and renumber only the rebased MicroVM decision and its citations to 0368.

Co-Authored-By: OpenAI Codex <codex@openai.com>
@JAORMX
JAORMX force-pushed the impl/native-kubernetes-execution branch from 77f456d to b7bf9fa Compare September 26, 2026 07:35
@JAORMX
JAORMX marked this pull request as ready for review September 26, 2026 07:42
@JAORMX
JAORMX merged commit 804298a into main Sep 26, 2026
74 of 79 checks passed
@JAORMX
JAORMX deleted the impl/native-kubernetes-execution branch September 26, 2026 09:54
This was referenced Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

e2e-live Run the live e2e suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant