[Bug]: Overlapping server lifecycles during a restart leave state.sqlite malformed and make the app unusable

[axelcursive]

Before submitting

• I searched existing issues and did not find an exact duplicate.
• I included the observed errors and server timeline.

Area

apps/server / persistence / server lifecycle

Summary

The T3 Code server initialized a new server lifecycle while an existing Cursor-backed turn was still producing and persisting activity.

The traces show that the new lifecycle began opening and writing to the same state.sqlite database before activity from the previous lifecycle had stopped.

Within approximately one second, SQLite began returning:

SQLITE(11) database disk image is malformed

The error now occurs in both the thread that was active during the restart and newly created threads. T3 Code can no longer append orchestration events, making the application unusable.

Observed timeline

Times below are UTC.

• 13:41:13.065 — a Cursor-backed turn starts.
• 13:44:30.039 — a new server initialization begins.
• 13:44:30.052 — the new lifecycle opens state.sqlite.
• 13:44:30.065 — it executes PRAGMA journal_mode = WAL.
• 13:44:30.218 — activity from the previously running turn is still inserted into orchestration_events.
• 13:44:30.238 — another activity append from the existing turn completes.
• 13:44:30.377–13:44:30.390 — additional activity from the existing turn is still persisted.
• 13:44:30.608 — the newly initialized server begins listening on port 3773.
• 13:44:30.691 — the new lifecycle inserts an orchestration event.
• 13:44:30.777 — startup session reconciliation records the previously active provider session as no longer available.
• 13:44:30.888 — the new server reports startup success.
• 13:44:31.122 — server queries begin failing with SQLITE(11) database disk image is malformed.

The previous and new server lifecycles therefore performed overlapping database activity during the restart window.

Restart context

The outer npx t3 process remained alive throughout the event. Its operating-system start time did not change.

The installed T3 Code version before and after the observed restart was 0.0.40. No server-update request or version change appears in the available server traces immediately before the new startup.

The logs confirm that a new internal server lifecycle started, but they do not expose what triggered that lifecycle restart.

Provider-session result

During startup reconciliation, the active turn was finalized with:

Provider session did not survive a server restart. Send a new message to continue.

The persisted provider runtime subsequently showed the Cursor session as stopped and the affected turn as completed with an error.

User-visible error

Attempting to stop or interact with the affected thread produces:

SQL error in OrchestrationEventStore.append:insert: SQLITE(11) database disk image is malformed

Creating a new thread and attempting to send its first message produces the same error:

SQL error in OrchestrationEventStore.append:insert: SQLITE(11) database disk image is malformed

The failure is therefore not limited to the turn that was active during the restart.

Database integrity

Read-only SQLite checks do not return ok.

Both PRAGMA quick_check and PRAGMA integrity_check report structural corruption, including:

• database pages referenced more than once;
• an orchestration event row ID reported out of order;
• btreeInitPage() returning error code 11;
• corruption involving the orchestration event table and its indexes.

Expected behavior

Restarting or reinitializing the server should not corrupt persisted state.

After a restart, existing threads should enter a recoverable state, and new threads should remain usable.

Actual behavior

The server database is structurally malformed.

All attempts to append new orchestration events fail, including attempts made from newly created threads. Existing and new agent work can no longer be started or continued.

Reproduction status

This was observed in a live remote-server session.

The exact action that triggered the internal server lifecycle restart has not yet been identified. The available traces establish the overlapping database activity and the first corruption errors, but do not contain the initiating restart event.

Environment

• T3 Code 0.0.40
• T3 Code Alpha desktop client on macOS
• Remote T3 Code server on Linux
• T3 Connect
• Cursor provider
• Local checkout mode

Impact

Blocks all agent work in the affected T3 Code environment.

Related issues

• Persisted state corruption can leave T3 Code unusable with no clear recovery path #961 — Persisted state corruption can leave T3 Code unusable with no clear recovery path
• [Bug]: Desktop starts a second backend against the background service database #6097 — Multiple server runtimes can access the same T3 Code database

[juliusmarminge]

Triage

Confirmed critical bug in server persistence / lifecycle. Not a Cursor-provider defect and not limited to the turn that was running.

The timeline is consistent with two server lifecycles writing state.sqlite in the same ~1s window. Current main (and 0.0.40) still does that: startup opens the DB, sets WAL, runs migrations, and only then binds HTTP. server-runtime.json is a discovery file, not a lock. busy_timeout only waits out SQLITE_BUSY; it does not make two owners safe.

WAL is supposed to serialize writers, so overlapping inserts alone should not produce SQLITE(11). The damaging part is a new lifecycle opening the same files (journal_mode / migrations / a new -shm mapping) while the previous lifecycle is still appending orchestration_events. HTTP can close first (port 3773 reusable) while those writes are still in flight. PRAGMA integrity_check failing with duplicate pages / btreeInitPage 11 matches that, and it is why new threads fail too.

The outer npx t3 start time not changing, and no update/version bump, means we still do not know what started the second lifecycle. The overlap itself is enough to treat this as a product bug. Candidates: Connect/service install beside a foreground server, SSH replace, a supervisor that treats a freed listen port as “dead”, or an in-process re-init. None of those paths currently refuse a second open.

Related

• [Bug]: Desktop starts a second backend against the background service database #6097 — same missing single-owner rule; observed as database is locked, not structural corruption. Not a duplicate.
• Persisted state corruption can leave T3 Code unusable with no clear recovery path #961 — no recovery path once persisted state is poisoned. Workaround remains: stop every server on that home, move state.sqlite + -wal + -shm aside, restart (history lost).
• fix(server): prevent duplicate servers for one state directory #9652 — in-flight exclusive ownership lock (open, conflicts with main). Also fix(server): refuse to start a second server against a live data directory #8442 / fix(server): enforce one owner per state directory #8960 / fix(server): refuse startup when another server owns the state dir #4420.

Next

Keep this issue as the corruption / restart-overlap tracker.

1. Exclusive state-dir ownership before opening state.sqlite; second start must fail closed.
2. Do not treat the listen socket as free for a replacement until SQLite connections are closed and the lock is released.
3. Log why server.startup ran again.

If the host is still available, fstype of the T3 home and whether a boot service / t3 connect install was running beside npx t3 would help pin the trigger. Not required to start the fix.