Repository navigation
fix(crews): J5 stream daemons start from the event store's high-water mark - #352
Conversation
… mark The Crew seat finish notifier, the Crew launch reporter, and the silence detector took their stream start point from MAX(sequence) of orchestration_v2_events, a table nothing writes since the upstream sync. The query returned 0, so every boot replayed the whole event history, and the Captain archive cascade riding the notifier's stream retired live Crews by replaying an old Captain archive. All three now read EventSinkV2.latestSequence(), the same store streamStoredEventsFrom reads. The Crew daemons go through readEventStoreHighWater, which logs and retries a failed read with capped backoff instead of falling back to 0. The silence detector's failed first cursor seed fails its init, which its lifecycle daemon already retries with backoff. The production J5 A2A layers get the orchestration runtime's own OrchestrationV2EventSinkLayerLive, so Effect memoizes one sink instance. Closes #349 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ression test The comment claimed the daemon-mode alert test waits until the test times out when the alert does not wake the worker. The worker's wake queue can hold a spare wake, so it passes without the fix; the wake-count test is the regression test. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Repository: Jacksondr5/j5code/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: Comment |
Jacksondr5
left a comment
There was a problem hiding this comment.
Posted by an AI agent on Jackson's behalf.
Approved. This is the right fix for #349. It reads the start point from the event store service itself, so the start point and the stream share one sequence space without repeating a table name. The wiring stays in J5-owned runtimeLayer.ts. 41 tests pass across the five affected files, and CI is green.
On your review-focus question: retrying forever with capped backoff is fine to keep. It's small, and it never falls back to 0, which is the actual bug. By the "repair beats edge-case machinery" principle, failing loudly would also have been acceptable, since a store that can't answer its latest sequence is broken anyway. Not a blocker.
This supersedes #353, which duplicated this fix and is now closed. Merge this before #315.
Problem
Three J5 background workers took their event-stream start point from
SELECT MAX(sequence) FROM orchestration_v2_events. Nothing has written that table since the upstream sync, because V2 events now live inorchestration_events, so the query returns 0 and every boot replays the whole event history (#349). Onj5/mainthat already lets the Captain archive cascade replay an old Captain archive on restart and retire live Crews while their Captain stays live. #315 would widen it to replayed unarchive, settle, and unsettle events, so this has to land before #315.What I changed
apps/server/src/j5/a2a/eventStoreHighWater.ts→readEventStoreHighWater: readsEventSinkV2.latestSequence(). On failure it logs and retries with capped backoff (250 ms up to 30 s), and never falls back to 0.CrewSeatFinishNotifier.ts→runDaemonandCrewLaunchReporter.ts→runDaemon: start from that helper. The raw queries are gone, and so is the launch reporter's unusedSqlClient.SilenceDetector.ts→initializeCursor: seeds its durable cursor fromlatestSequence(). A failed read fails the seed, and the lifecycle daemon already retries that step.runtimeLayer.ts: the J5 runtime layers getOrchestrationV2EventSinkLayerLive. It's the same layer object the orchestration runtime uses, so there's one sink instance.Why this shape
EventSinkV2rather than the lower-level store:ThreadManagementService.streamStoredEventsFromstreams from the sink, so the start point and the stream share one sequence space. It's also whatQueuedRunWatchdogalready uses.Invariants
Surfaces
Out of scope
Upgrade and data
None. The next boot after this lands starts each stream at the current high-water mark.
Verification
apps/server,apps/web, andpackages/contractstypecheck (exit 0).vp test run apps/server/src/j5: 1,061 passed, 1 skipped.runtimeLayer.test.ts: 47 passed.latestSequence, including after a failed read.Review focus
readEventStoreHighWater, and whether a daemon should ever give up.Closes #349
Claude Opus 5.5 via Claude Code in J5 Code
🤖 Generated with Claude Code