Skip to content

fix(grok): retire crashed sessions before deliberate resume - #923

Merged
rynfar merged 2 commits into
pylonfrom
upstream/2026-09-30-grok-crash
Sep 30, 2026
Merged

rynfar merged 2 commits into
pylonfrom
upstream/2026-09-30-grok-crash

Conversation

@rynfar

@rynfar rynfar commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

When the Grok ACP subprocess crashed, Pylon kept its dead context registered and sent the next prompt to it. This retires the context on ConnectionTerminated: new sends are refused immediately, any active turn settles as failed, and cleanup emits an error exit. A deliberate restart uses the saved ACP cursor through session/load.

Cleanup runs in the adapter's owning scope under the existing thread lock. The context identity check prevents a queued teardown from touching a replacement session; the notification consumer does not wait on teardown that interrupts itself. Pylon incarnation tags and native task maps remain intact, including completed/ambiguous task identities. Graceful Stop and ordinary prompt errors keep their existing behavior. No automatic retry is introduced.

Upstream accounting

Complete three-file port of 050cfad04f673fb092630614b8f7e82000b43ea5, from the separately frozen successor bound 0fcd5f90611451cca842689faea53b5450c022da, owning cycle #865. The attributed cherry-pick includes the process-crash mock and receipt-driven retry regression. Conflict resolution retains Pylon session-incarnation fields and passes the owner to settlePromptInFlight.

Pylon test adaptations scope the mock launcher/request log to disposable filesystem state, assert exactly one failed completion plus an error exit with the crashed incarnation, and assert that the resumed session has its new incarnation. The protocol log verifies one session/new and one deliberate session/load using the original cursor.

Verification

  • 63 focused Grok adapter/driver and ACP runtime tests pass; server package typecheck and scoped lint/format/diff checks pass.
  • The crash regression fails against the pre-port Pylon adapter because hasSession still reports true; restored code passes. Tests wait on exit receipts, not sleeps.
  • 17 focused ingestion regressions also pass for background work, incarnation fences, stale session events and durable Stop. Pylon ingestion consumers inspected for stale-incarnation rejection, Stop barriers and background-liveness cleanup. Existing native task state and session-scope ownership retained.
  • No live Grok account, browser/device verification, release or install. No wire/schema or client changes. This PR is a pending port, not completion of the upstream audit.

Model: GPT-6. Harness: Codex in Pylon.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M labels Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire — 14.0 KiB — 15.1 KiB ✅
Codex Thread snapshot wire — 7.3 KiB — 7.3 KiB ✅
Codex Live turn WebSocket wire — 6.7 KiB — 7.8 KiB ✅
Codex Live turn WebSocket decoded — 58.0 KiB — 66.4 KiB ✅
Codex Live turn messages — 9 — 21 ✅
Claude Total thread wire — 14.0 KiB — 15.1 KiB ✅
Claude Thread snapshot wire — 7.3 KiB — 7.3 KiB ✅
Claude Live turn WebSocket wire — 6.7 KiB — 7.8 KiB ✅
Claude Live turn WebSocket decoded — 58.8 KiB — 66.4 KiB ✅
Claude Live turn messages — 8 — 21 ✅

Baseline: unavailable · PR result: d361680 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 115.7 KiB
  • Claude decoded thread snapshot: 116.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

saphid and others added 2 commits September 30, 2026 09:03
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: SAPHID <saphid@users.noreply.github.com>
(cherry picked from commit 050cfad04f673fb092630614b8f7e82000b43ea5)
@rynfar
rynfar force-pushed the upstream/2026-09-30-grok-crash branch from c32d4b8 to d361680 Compare September 30, 2026 15:03
@rynfar
rynfar merged commit da77592 into pylon Sep 30, 2026
17 checks passed
@rynfar
rynfar deleted the upstream/2026-09-30-grok-crash branch September 30, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants