Skip to content

[think] Cold-start submitMessages drain executes before onStart initializes session — "Cannot read properties of undefined (reading 'appendMessage')" #1967

Closed

Description

@cgrdavies

Describe the bug

Calling the submitMessages() RPC on a cold (hibernated/evicted) Think agent durably records the submission, but the post-accept drain can execute it before the agent's onStart has assigned this.session (this.session = await this.configureSession(baseSession)). The turn then crashes in _appendMessageToHistory with Cannot read properties of undefined (reading 'appendMessage'), and the submission is marked terminally failed — the durable row survives, but the turn it represents is lost (a retry under the same idempotency key is rejected as a duplicate; a retry under a fresh key hits the same cold-start window).

Warm agents — any instance where a chat connection or other code path already ran onStart in the same lifetime — process the identical submission fine, which makes this intermittent and easy to misdiagnose.

To Reproduce

  1. A Think agent DO (stock or custom configureSession).
  2. Ensure the DO is cold: no WebSocket/chat connection, evicted or never woken since deploy.
  3. From another Worker, call stub.submitMessages([message], { idempotencyKey, metadata }) over a DO stub — a "completion callback"-style programmatic turn.
  4. Observe [Think] Submission failed … "Cannot read properties of undefined (reading 'appendMessage')" and a terminally-failed submission row.

Sequence (published dist, v0.12.1 line refs): submitMessages accepts → _startSubmissionDrain() → keepAliveWhile(() => this._drainSubmissions()) (think.js ~5206) → _runSubmission → turn execution → _appendMessageToHistory (~1311) dereferences this.session, which the wrapped onStart (~934) has not assigned yet.

Expected behavior

submitMessages (or the drain it triggers) should await the agent's internal readiness — session initialization — before executing a submission. Callers of a public RPC shouldn't have to know about onStart timing. Notably, the alarm path Think schedules via this.schedule(0, "_drainThinkSubmissions") re-enters through the normal agent lifecycle, but the immediate keepAliveWhile drain does not wait for it.

Environment

  • @cloudflare/think 0.12.1
  • The same cross-worker callback pattern ran reliably for us on 0.8.8 under production traffic; after upgrading to 0.12.x we see daily failures whenever the target DO is cold.

Workaround

Subclass override that forces initialization before delegating:

override async submitMessages(...args: Parameters<Think['submitMessages']>) {
  await this.ensureInitialized() // awaits __unsafe_ensureInitialized() and asserts this.session
  return super.submitMessages(...args)
}

Impact

Any "durable completion hook" pattern (cf. #1752) that wakes an idle agent via submitMessages silently loses events whenever the target agent is cold — for us that was workflow-completion and delegated-sub-agent-completion notifications in production.

Activity

  1. tylergibbs1 commented on Jul 25, 2026

    @tylergibbs1
    Contributor

    I’d like to work on this. I’ll add a cold-start regression test around submitMessages(), make the immediate submission drain wait for Think’s session initialization before executing queued work, and include the required @cloudflare/think changeset. I’ll keep the change scoped to readiness ordering and preserve the existing durable/idempotent submission semantics.

  2. tylergibbs1 commented on Jul 25, 2026

    @tylergibbs1
    Contributor

    Update: while preparing the fix, I found Cloudflare’s existing feat/rpc-self-init branch (commits 766f1614 / 938a63bd), which makes user-defined cold RPC entry surfaces await onStart() and appears to cover the submitMessages() initialization race described here. I’m going to avoid opening a duplicate implementation. Is #1967 intended to be resolved by that branch, or is a Think-specific guard still expected?

  3. threepointone commented on Aug 3, 2026

    @threepointone
    Contributor

    we don't accept contributer PRs @tylergibbs1

  4. self-assigned this
    on Aug 5, 2026
  5. threepointone commented on Aug 5, 2026

    @threepointone
    Contributor
  6. cgrdavies commented on Aug 5, 2026

    @cgrdavies
    Author

    Thanks @threepointone and @cjol !

  7. mattzcarey commented on Sep 10, 2026

    @mattzcarey
    Member

    Closed as fixed: the immediate cold-start drain was removed in #1860; submissions now drain via the alarm path, which initializes the agent first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions