[Feat] Durable diagnostic run events - #140
Merged
Merged
Conversation
Sandbox logs do not survive the sandbox and Modal exposes none at all, so runtime facts worth a post-mortem keep dying with the machine. This adds a diagnostic rail on the existing task_run_events audit table: - a `diagnostic` run event type, recorded by workers through the existing sdk.taskRuns.recordEvent path (no schema change), - a worker-side recorder that is observer-only: fire-and-forget, never throws into the caller, caps every string, and scrubs token-shaped values and credential assignments before anything leaves the process, - a `tasks.runEvents` web tRPC reader so per-task events are queryable without database access. Nothing emits through the recorder yet; the OpenCode exit certificate lands separately on top of this rail. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Reviewed this PR. The diagnostic rail is well-scoped, observer-only, and the never-throw / scrubbing guarantees are backed by tests. 3 non-blocking things to consider (1 medium, 2 low). See task
|
This was referenced Jul 10, 2026
daniel-lxs
added a commit
that referenced
this pull request
Jul 10, 2026
…icate (#147) * [Fix] Address review findings on the diagnostic rail and death certificate Review follow-up for #140 and #145, which merged before their bot reviews were read: - Deliberate harness terminations (reconnect, restart, teardown) are no longer certified as unexpected exits: ReconnectableHarness marks the subprocess before killing it, and certification skips marked exits, which also silences the Sentry alert that fired on every routine reconnect. - The death tail no longer depends on the startup URL wait leaving its line listeners attached: dedicated readers feed the ring for the whole subprocess lifetime. - The run-events reader returns the newest events under its limit (descending query, reversed to chronological), so a busy task cannot push the diagnostics it exists to surface out of the window. - A caller-supplied details.kind can no longer override the recorder's classification. - Hash-shaped values (git SHAs, digests) keep an 8-character prefix when redacted, preserving their evidentiary value without exposing credential-length material. - The memory reading is named and documented as what it is: the sandbox observed just after the exit, by the surviving worker process — the exit code/signal remains the authoritative OOM signal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Mark expected exits only while the subprocess is alive A crash surfaces as a disconnect first, and the disconnect cleanup then kills the already-dead process; marking it there raced the exit certification and could suppress the certificate for exactly the crash the rail exists to record. The mark now applies only to a process that is still alive at kill time, so deliberate teardowns stay quiet and self-inflicted deaths stay certified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 1 of the observability pair (PR 2, the OpenCode exit certificate, stacks on this). Motivation: sandbox logs die with the sandbox and Modal exposes none at all, so the last two incident investigations ended at "leading suspect" instead of proof.
What
A diagnostic rail on the existing
task_run_eventsaudit table — no schema change:diagnosticadded torunEventTypes; workers write through the existingsdk.taskRuns.recordEventpath.apps/worker/src/run-task/diagnostic-events.ts).tasks.runEventsweb tRPC reader (mirrorstasks.messageEnvelopesauth) so per-task events are queryable without DB access.Nothing emits yet — the first producer (OpenCode death certificate) is the follow-up PR, kept separate so this rail can be reviewed on its own.
Guarantees (per the requirements)
*_PASSWORD=/*_TOKEN=style assignments are redacted (tests cover each). Events land in the same org-scoped DB that already stores full task transcripts, behind the same auth.Tests
6 new tests: scrubber coverage (keys, JWTs, assignments incl. prefixed names, plain-text passthrough), capped+scrubbed recording, never-throws on persistence failure and on cyclic details. Typecheck clean across types/sdk/worker/web; knip clean.
Not auto-merged — ready for review.
🤖 Generated with Claude Code