Repository navigation
test(orchestration): distinguish the heartbeat straggler guard from the row's initial null - #15869
Conversation
…he row's initial null The straggler case never gave the row a heartbeat before settling it, so last_heartbeat_at was null from creation and the expected value equalled the initial state. That did catch outright removal of recordHeartbeat's `AND status = 'dispatched'` filter, but it conflated "guard held" with "completeDispatch cleared the column" and with "recordHeartbeat is inert" — both of those regressions kept toBeNull() green. Seed a heartbeat while the row is still dispatched so the expectation is a timestamp: null now means the column was cleared, '2026-05-04...' means the guard is gone. Carried from #15165.
…es too recordHeartbeat filters on status='dispatched' exactly, so it also ignores stragglers on rows settled by failDispatch. Only the completed path was covered, so widening the filter to `status != 'completed'` passed the suite. Co-authored-by: Tauri-EPO <enrico.pin@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
💤 Files with no reviewable changes (1)
Included review availability: Your plan provides up to 10 included reviews per hour; 4 remain after this review. 📝 WalkthroughWalkthroughAdded a Vitest suite that verifies late heartbeats remain ignored after dispatch completion and failure. The suite initializes and cleans up an in-memory 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
db.test.ts landed at exactly 800 effective lines — the max-lines ceiling for **/*.test.ts — which closed the file to any further addition until #14889 splits it, and forced the failDispatch sibling of this same invariant into dispatch-failure-idempotency.test.ts where its name does not fit. Move both straggler cases into db-heartbeat-straggler-guard.test.ts. The assertions and the seeded-heartbeat fixture are unchanged; db.test.ts drops to 790 effective lines (10 of headroom) and this PR no longer has a merge-order dependency on #14889. Co-authored-by: Tauri-EPO <enrico.pin@gmail.com>
|
Thanks for carrying this through, and for the correction on the commit subject — you are right, and I was overstating it. "It can never fail" describes injection C only; removing the The Noted on the I will close #15165 once this lands. |
…he row's initial null (stablyai#15869) Co-authored-by: Tauri-EPO <enrico.pin@gmail.com> (cherry picked from commit 8c1e6ad)
…he row's initial null (stablyai#15869) Co-authored-by: Tauri-EPO <enrico.pin@gmail.com>
…he row's initial null (stablyai#15869) Co-authored-by: Tauri-EPO <enrico.pin@gmail.com>
…he row's initial null (stablyai#15869) Co-authored-by: Tauri-EPO <enrico.pin@gmail.com>
Supersedes #15165 by @Tauri-EPO — carried over so it can run full CI (fork PRs only run
track-community-prin this repo). Original authorship preserved in the commit.ELI5
A test claimed "a heartbeat that arrives after the job already finished is ignored." It proved that by checking the heartbeat column ended up empty — but the column starts empty and the test never put anything in it. So "the guard correctly ignored the late heartbeat" and "something wiped the column" looked identical to the test, and so did "
recordHeartbeatdoes nothing at all."This PR writes a real heartbeat first, so the assertion compares against a timestamp instead of against the row's starting state. A second test does the same for dispatches that end in failure rather than success — a gap that let a whole class of regression through unnoticed.
Test-only. No product code changes.
Background: what the straggler guard is
A dispatch is one attempt at running a task on a worker terminal. While it runs, the worker sends a heartbeat every 5 minutes; each one stamps
dispatch_contexts.last_heartbeat_atviarecordHeartbeat(src/main/runtime/orchestration/db/dispatch-context/dispatch-completion.ts:51-58):That
AND status = 'dispatched'is the straggler guard. A straggler is a heartbeat that arrives after the dispatch has already settled —completed(dispatch-completion.ts:9-16), orfailed/circuit_broken(failDispatch,dispatch-completion.ts:76-105; the status iscircuit_brokenoncefailure_countreachesDISPATCH_CIRCUIT_BREAK_FAILURES = 3,dispatch-circuit-breaker.ts:2). Stragglers are normal, not exotic: a heartbeat is a message in flight, and the worker can settle while one is still in transit — more so over an SSH/federation relay, where the delivery leg is long.The failure it protects against is on the read side.
getStaleDispatches(dispatch-completion.ts:60-74) is what notices a wedged worker:Its only caller is
warnStaleDispatches(coordinator-task-dispatch.ts:17-26), which tells the coordinatorworker <handle> on task <id> has not sent a heartbeat in ~10 min. The threshold is deliberately the documented cadence × 2 (coordinator-task-dispatch.ts:13-14), and it warns rather than auto-fails (:16). Solast_heartbeat_atis the sole liveness signal, and the warning is the only thing that surfaces a hung worker to a human. Corrupt that column and a genuinely hung dispatch simply never gets reported — the task sits there looking alive, holding its worker slot, forever. There is no second detector behind it.Why the DB-level guard has to exist even though callers check too.
recordHeartbeathas three callers. Only one pre-checks status:lifecycle-reconciliation.ts:170, guarded by:150-157(suppresses heartbeats for non-dispatcheddispatches). The other two do not:db/federation/federation-relay-import.ts:92— heartbeats imported from a federated relay; guarded only by!duplicate.db/legacy/legacy-lifecycle-operation.ts:141— the legacy lifecycle path.On those two paths the SQL status filter is the only thing standing between a relayed straggler and the liveness column. That is what the test in question is supposed to be guarding.
What Changed
src/main/runtime/orchestration/db-heartbeat-straggler-guard.test.tssrc/main/runtime/orchestration/db.test.tsmain:657-667)The new file shares one fixture,
seedHeartbeatedDispatch()(db-heartbeat-straggler-guard.test.ts:12-19), which records a heartbeat of'2026-05-03T00:00:00.000Z'while the row is stilldispatched(:17) — that seed line is the entire point of the change. Two cases then settle the row and fire a straggler at'2026-05-04…':ignores a straggler heartbeat after the dispatch completed(:23-30) — the migrated case.ignores a straggler heartbeat after the dispatch failed(:32-39) — new.recordHeartbeat updates last_heartbeat_at on dispatched rowsstays indb.test.ts; nothing else moved. Test counts:db.test.ts61 → 60, new file 2, so 62 total — one case relocated, one added.The assertion, before and after
Before (
db.test.tsonmain, lines 657-667):After (
db-heartbeat-straggler-guard.test.ts:23-30):Why the old one was vacuous.
last_heartbeat_atisNULLfrom row creation, and the old test never wrote to it, sotoBeNull()was an assertion about the row's initial value. It therefore held for three mutually exclusive reasons that the test could not tell apart:completeDispatchcleared the column on its way through;recordHeartbeatwas inert for every row, guard or no guard.Only (1) is correct behavior. (2) and (3) are product bugs that would have shipped with this test still green — and (2) is a bug in the exact column the stale detector reads. The new expected value is a timestamp, which separates them:
'2026-05-03…'means the guard held,nullmeans something cleared the column,'2026-05-04…'means the guard is gone.One correction to the issue's framing. #15164 / #15165 describe the old assertion as one that "can never fail." That is overstated, and I checked rather than repeating it — see mutation C below, where removing the status filter outright does turn the old assertion red, because the straggler write leaves
'2026-05-04…'wherenullwas expected. The accurate claim is narrower: the old assertion could not distinguish the guard working from the column being cleared, and it had no coverage at all for settled-as-failedrows. The commit subject states that weaker, true claim.Why the second test exists
The filter matches
status = 'dispatched'exactly, so it also rejects stragglers on rows settled asfailedorcircuit_broken. Only thecompletedpath was ever covered. That leaves a specific, very plausible regression uncaught: someone rewrites the guard asstatus != 'completed'— which reads like a faithful paraphrase of "ignore heartbeats after the dispatch finished" and is wrong, becausefailedandcircuit_brokenare also finished. Under that change the entire suite stayed green onmain. It does not any more.Why a new file rather than editing in place
db.test.tsonmainmeasures 799 effective lines against the 800 ceiling for**/*.test.ts(.oxlintrc.json:157-160,skipBlankLines+skipComments). Adding the one-line seed in place would have put it at exactly 800 — legal, but with zero headroom, closing the file to any further test until #14889 splits it, and forcing thefailDispatchsibling to live in an unrelated file whose name does not describe it. Amax-linesdisable is not an option (AGENTS.md hard rule).Moving both cases out instead solves all three problems at once, and the numbers are in the right direction — see Known limitations for the re-measured figures.
Linked Issue
Fixes #15164
Visual Proof
N/A— test-only change. No product code, no rendered UI, no user-visible behavior.Testing
macOS (Darwin 25.3.0), Node via repo toolchain. The contributor reports the original assertion green on Windows 11 / Node 24.19.0.
Baseline, both touched files:
Regression-injection proof
A test that could not distinguish outcomes is only provable by making it fail, so I mutated the product code and ran the new assertion and the old one side by side against the same mutation. The old assertion was restored verbatim into a scratch file for the duration; all mutations and the scratch file were reverted afterwards, and
git statusis clean (the product file is untouched in this PR — the diff is 2 test files).Mutation A — the realistic straggler bug. Guard widened to
status != 'completed'indispatch-completion.ts:55:The
1 passedtest file is the old assertion — green on the bug. This is the coverage gap made concrete: a relayed straggler now overwrites the liveness stamp on afailed/circuit_brokenrow, and the pre-PR suite says everything is fine.Mutation B — the confound.
completeDispatch(:13) also setslast_heartbeat_at = NULL, the guard left intact:Again the old assertion passes —
nullwas exactly what it expected. This is reason (2) above: the heartbeat history the stale detector depends on is being destroyed on completion, and the test written to guard that column is satisfied by the destruction.Mutation C — honesty check.
AND status = 'dispatched'removed entirely:All three fail, old assertion included. That is why the "can never fail" framing above is corrected rather than repeated.
Post-mutation restore:
git statusclean,62 passed (62)again.Lint and format
Not run locally:
pnpm typecheck, fullpnpm lint,pnpm build, full suite — CI covers them (these OOM on this machine).Review
3d39601by Tauri-EPO). No network calls, noprocess.env/ fs / shell /eval/child_process, no dependency, lockfile, workflow, or installer changes, no embedded secrets. Enumerated every codepoint >127 in the patch — the only non-ASCII character was U+2014 em dash in a comment; no zero-width, bidi, or BOM characters (no Trojan-Source vector). Thelast_heartbeat_at = NULLinjection described above was a local experiment and is not in the diff; confirmed againstgh pr diff.TEXT. No wall clock, no timezone, no path separators, no line-ending sensitivity, nometaKey.live/unverifiable/exitedvocabulary is not touched. (The federation relay path is described above as motivation, not modified.)recordHeartbeat/completeDispatch/failDispatchmethods against an in-memory DB, so there is no wire or migration surface.:memory:database, no I/O.Agent skill upstream boundary
docs/reference/agent-skill-sharing-upstream-boundary.mdand copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.Notes
Deviations from #15165 as authored: the 3-line
// Why:block was collapsed to one line per AGENTS.md, the commit subject was rewritten to the accurate claim (the old assertion was weak, not incapable of failing), thefailDispatchsibling case was added, and both cases were relocated intodb-heartbeat-straggler-guard.test.ts. The contributor's authorship is preserved on the original commit and carried as aCo-authored-bytrailer on the relocation commit.The assertion also switched from
getDispatchContext(task.id)togetDispatchContextById(dispatchId). Minor, but deliberate: the former returns the newest row for the task (dispatch-context-store.ts:119-126,ORDER BY rowid DESC LIMIT 1), so it would follow a retry onto a different row; the latter pins the exact dispatch the straggler was addressed to (:128-134), which is what the invariant is about.Known limitations / follow-ups
db.test.tsremains large — but this PR frees budget rather than consuming it. Re-measured on both sides with the same method (append N synthetic code lines, read the count oxlint reports, subtract N):mainBoundary-checked rather than extrapolated: on this branch +10 synthetic lines lints clean and +11 errors with
File has too many lines (801); onmain+1 lints clean and +2 errors at 801. So merging this returns ~9 lines to a file that currently has one line of room left. test(orchestration): split dispatch-liveness tests out of db.test.ts #14889 (also @Tauri-EPO) splits the dispatch-liveness cases out and is the durable fix; it is not folded in here because that is a different, larger change, and this PR no longer depends on its merge order.circuit_brokenhas no straggler case of its own.failDispatchtransitions tocircuit_brokenafterDISPATCH_CIRCUIT_BREAK_FAILURES(3) consecutive failures, and thestatus = 'dispatched'filter excludes it for the same reason it excludesfailed, but no test pins it. Driving a row to that status needs a multi-failure fixture that would outgrow this PR.The
§5.3.4reference in therecordHeartbeatcomment does not resolve to anything indocs/today. Left as-is (untouched product code), noted here for whoever maintains that comment.Checklist
N/Awith reason