Skip to content

feat(engine): daemon-minted event sequencing and a bounded live journal - #509

Open
Zerlight wants to merge 4 commits into
ruocheng/code-629from
ruocheng/code-630
Open

feat(engine): daemon-minted event sequencing and a bounded live journal#509
Zerlight wants to merge 4 commits into
ruocheng/code-629from
ruocheng/code-630

Conversation

@Zerlight

@Zerlight Zerlight commented Sep 7, 2026

Copy link
Copy Markdown
Member

Summary

Phase 1 of CODE-627 — Conversation turn graph & immutable attachment store. Linear: https://linear.app/arcbox/issue/CODE-630/featengine-daemon-minted-event-sequencing-and-a-bounded-live-journal

Stack: #508this PR (ruocheng/code-630, base ruocheng/code-629) ← #510. Merge bottom-up; this PR's diff is only its own commits.

The daemon owns event ordering: a persisted per-session epoch bumped on every run launch and boot, seq minted onto the agent.event broadcast path, a bounded per-session live journal (byte and event caps, truncation marker), and run-gated session-scoped events so a replaced adapter's stragglers never paint the session.

Commits

  • feat(engine): persist a per-session event epoch
  • feat(engine): add the bounded per-session live event journal
  • feat(engine): mint (epoch, seq) onto the agent.event broadcast path
  • fix(engine): make the launch epoch durable before minting and flag future watermarks

Verification

Every commit passed pnpm check:ci and pnpm test at its own tip; the stack tip (1d942a62; the same tree as the originally gated 73fc5ff2 plus the two review fixes below on 628/629) is at pnpm check:ci 0 errors, pnpm test 3408 passed / 1 skipped. Adversarial reviewers (one per axis, isolated read-only worktrees) reviewed the branch; each P1/P2 was reproduced with a failing test or a probe step before its fix — the round-by-round record is in the Linear issue's comments.

Checklist

  • pnpm check:ci and pnpm test both pass (no Rust changes)
  • I ran the affected surface and observed the change working — engine exercised through its harness tests and the dev-mock parity suite; the live-daemon run against real providers is recorded per issue in Linear (the claude edit path still needs a paid turn, opencode its binary)
  • Wire: optional envelope fields added under the still-unshipped wire 80
  • New code and assets are my own work
  • Docs and comments are updated where behavior changed (AGENTS.md and module docs in this branch)

@pullfrog

pullfrog Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Your Claude subscription has hit its usage limit. It resets at 11:30am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

Pullfrog  | Rerun failed job ➔View workflow run | via Pullfrog | Using Claude Opus𝕏

@linear-code

linear-code Bot commented Sep 7, 2026

Copy link
Copy Markdown

CODE-630

@greptile-apps

greptile-apps Bot commented Sep 7, 2026

Copy link
Copy Markdown

Greptile Summary

This PR introduces persisted per-session event epochs, daemon-minted sequence numbers, stale-run gating, and bounded in-memory journals for live conversation events.

  • Adds the event_epoch database column and carries it through session records and stores.
  • Centralizes stamped agent.event emission and records each event in a bounded per-session journal.
  • Bumps epochs across launches and daemon boots to order replacement-run events above stragglers.
  • Adds sequencing, truncation, lifecycle, and persistence tests.
  • The review found incomplete stale-run isolation, plus two journal consistency and lifecycle concerns.

Confidence Score: 4/5

The PR should not merge until stale replaced adapters are prevented from mutating current session-global state; the remaining journal concerns are non-blocking but should also be hardened.

A replaced adapter can still overwrite the current session title or generate current-session notifications through event types omitted from the stale-run gate. Journal deletion and send-failure ordering also leave narrower consistency risks.

Files Needing Attention: packages/host/engine/src/session/session-event-processor.ts, packages/host/engine/src/session/orchestrator.ts

Important Files Changed

Filename Overview
packages/host/engine/src/session/session-event-processor.ts Centralizes event stamping and journaling, but stale turn events retain session-global side effects and failed sends can diverge the journal from the wire.
packages/host/engine/src/session/orchestrator.ts Integrates journals with LiveSession lifecycle; the final unguarded drop in concurrent deletion can remove a replacement run's journal.
packages/host/engine/src/conversation/live-journal.ts Implements byte/event-bounded retention, watermark tails, and gap detection with comprehensive cap and ordering tests.
packages/host/engine/src/session/session-record-registry.ts Adds boot and launch epoch increments plus an awaited persistence operation for relaunches.
packages/host/engine/src/session/lifecycle-service.ts Initializes epochs for new records and durably flushes bumped epochs before relaunching adapters.
apps/daemon/src/session-store.ts Correctly maps the new event epoch between persisted rows and session records.
apps/daemon/src/db/schema.ts Adds the non-null, default-zero event epoch column corresponding to the generated migration.
packages/foundation/schema/src/model/conversation.ts Adds a straightforward lexicographic comparator for epoch and sequence watermarks.

Sequence Diagram

sequenceDiagram
    participant A as Agent adapter
    participant P as SessionEventProcessor
    participant R as SessionRecordRegistry
    participant L as LiveSession
    participant J as Live journal
    participant T as Transport

    A->>P: AgentEvent
    P->>R: isCurrentRun(sessionId, runId)
    alt stale session-scoped event
        P-->>A: Drop event
    else accepted event
        P->>L: nextSeq()
        L-->>P: (epoch, seq)
        P->>J: append stamped event
        P->>T: send agent.event
    end
Loading

Reviews (1): Last reviewed commit: "fix(engine): make the launch epoch durab..." | Re-trigger Greptile

Comment thread packages/host/engine/src/session/session-event-processor.ts
Comment thread packages/host/engine/src/session/orchestrator.ts
Comment thread packages/host/engine/src/session/session-event-processor.ts
Copilot AI lite review requested due to automatic review settings September 7, 2026 06:54

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greptile has paused reviews on this repository — it used its 100 free open-source review credits for this billing period. Reviews resume automatically on September 23. To continue before then, an organization admin can keep reviews running past the free credits — those bill as normal usage.

@pullfrog

pullfrog Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Your Claude subscription has hit its usage limit. It resets at 11:30am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

Pullfrog  | Rerun failed job ➔View workflow run | via Pullfrog | Using Claude Opus𝕏

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core engine event ordering/journaling and persists new sequencing state that affects wire semantics and session lifecycle behavior.

Pull request overview

Implements daemon-minted ordering for agent.event frames by introducing a per-session eventEpoch and per-adapter monotone seq, plus a bounded in-memory per-session live journal that records exactly what is sent over the wire (including attribution fields).

Changes:

  • Add durable eventEpoch to session records and bump it on run launches (and in-memory on boot), ensuring replacement adapters mint above stale runs.
  • Stamp every agent.event wire frame with (runId, turnId?, epoch, seq) at a single “stamped exit” and append the same stamped events into a bounded live journal.
  • Add tests covering wire stamping, epoch durability before minting, stale-run gating, and live-journal boundedness/gap semantics.
File summaries
File Description
packages/host/engine/src/session/session-record-registry.ts Adds eventEpoch bumping semantics and an awaited flush() for launch-time durability.
packages/host/engine/src/session/session-input-dispatcher.ts Threads LiveSession into event broadcasting/rejection so stamps/journaling are consistent.
packages/host/engine/src/session/session-event-processor.ts Centralizes stamping (epoch, seq) + journaling for every agent.event frame and gates session-scoped stale-run events.
packages/host/engine/src/session/orchestrator.ts Wires in ConversationLiveJournals, drops journals on teardown/delete, and updates broadcast call sites.
packages/host/engine/src/session/live-session.ts Adds per-adapter seq minting and carries launch-captured epoch.
packages/host/engine/src/session/lifecycle-service.ts Ensures launch path awaits a durable eventEpoch save before constructing a LiveSession.
packages/host/engine/src/engine.ts Instantiates and injects ConversationLiveJournals into the session subsystem.
packages/host/engine/src/conversation/turn-service.ts Exposes runningTurnId() for event attribution on the broadcast path.
packages/host/engine/src/conversation/live-journal.ts Introduces the bounded per-session live journal and registry (open/get/drop).
packages/host/engine/src/tests/session-record-registry.test.ts Adds coverage for epoch bumping on launch and in-memory boot bump persisted on next launch.
packages/host/engine/src/tests/session-event-sequencing.test.ts New: validates wire stamping, epoch bumps across relaunch, durable pre-mint save ordering, and stale-run behavior.
packages/host/engine/src/tests/engine-turn-tracking.test.ts Updates test fixtures for new eventEpoch field.
packages/host/engine/src/tests/engine-session-records.test.ts Updates session record fixtures to include eventEpoch.
packages/host/engine/src/tests/engine-session-lifecycle.test.ts Updates LiveSession construction sites for new constructor signature.
packages/host/engine/src/tests/engine-session-input.test.ts Relaxes assertions to account for optional new envelope fields on agent.event.
packages/host/engine/src/tests/conversation-live-journal.test.ts New: tests watermark merge rules, caps, and tailAfter() gap semantics.
packages/foundation/schema/src/model/session/record.ts Extends SessionRecordSchema with durable eventEpoch.
packages/foundation/schema/src/model/conversation.ts Adds compareConversationWatermarks() helper for lexicographic (epoch, seq) ordering.
packages/client/workbench/src/mock/dev-mock-host.ts Updates dev-mock session record creation for new schema field.
apps/daemon/src/session-store.ts Persists/loads eventEpoch in the SQLite session store mapping.
apps/daemon/src/db/schema.ts Adds event_epoch column to sessions table schema.
apps/daemon/src/tests/session-store.test.ts Extends store roundtrip test to include eventEpoch.
apps/daemon/drizzle/meta/0013_snapshot.json Updates Drizzle snapshot metadata to include the new column.
apps/daemon/drizzle/meta/_journal.json Registers the new migration entry.
apps/daemon/drizzle/0013_famous_randall.sql Adds migration to introduce event_epoch to sessions.
Review details
  • Files reviewed: 25/25 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/host/engine/src/session/session-record-registry.ts
Copilot AI review requested due to automatic review settings September 7, 2026 07:25
@pullfrog

pullfrog Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Your Claude subscription has hit its usage limit. It resets at 11:30am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

Pullfrog  | Rerun failed job ➔View workflow run | via Pullfrog | Using Claude Opus𝕏

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@Zerlight

Zerlight commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

Review round (2026-09-07). Verified every Copilot and Greptile finding against the branch.

Fixed — ed32870b, on the top of the stack (#517) because delete() is unchanged from here upward: the orchestrator marks a session as deleting for the whole of delete() and refuses to install a live run for it meanwhile, so a resume admitted during the delete's store waits can no longer leave an orphan live session or lose its journal to the final drop. Serializing on the session permit was ruled out: a resume parked on the runtime probe holds it, and delete must win immediately in that case (the runtime-gating test pins it).

Declined, with the reasoning on each thread: the stale-run gate's scope (teardown unsubscribes the adapter before any replacement run exists, so a dead run's events never reach the processor), journal-before-send ordering (the hub's send is best-effort per connection and never throws), and the boot epoch bump (in-memory by design: nothing mints before a launch, and every launch persists).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants