Repository navigation
feat: keep thread metadata and event logs in a Turso database - #594
Merged
Merged
Conversation
sessions.json and every <id>.jsonl move into tcode.db, owned by SessionStore. Metadata rows hold each project's and session's JSON as before; event rows hold each raw log line, delimiter included, so export is byte-identical. One writer connection commits the store writer's queued writes as one transaction, with synchronous=FULL and, on Apple, fullfsync, so an acknowledged write survives a crash; a failed commit is reported for every write in it and a later flush cannot certify it. An imported or forked thread commits its events and metadata together. The first start builds tcode.db.migrating from the index and every log in the data directory, verifies each thread byte for byte and the metadata field for field, marks the database complete, checkpoints, syncs and renames it, then moves the sources into legacy/. An interrupted migration starts over; a complete database is never deleted or overwritten, and a corrupt one is reported with the sqlite3 CLI as the salvage path. A lock file owned for the host's lifetime keeps a second host out of the data directory; a relaunch waits for the outgoing one. Closes #520 Closes #523
…quit The migration is its own cancellable step with progress, run under the data directory's ownership lock before the host starts. The desktop opens a modal dialog with a progress bar and a Quit button first and starts the kernel and shell in the same process once it completes; Quit, ⌘Q or closing the window stops the migration at its next chunk, discards the staging files and exits. Headless prints progress and treats SIGINT and SIGTERM as cancel. No source file is written, renamed or removed before the staged database is published; an unparseable sessions.json is no longer renamed during the build but archived unchanged afterwards.
Tryanks
enabled auto-merge
October 6, 2026 06:47
The export test snapshotted every file in the data dir to prove an export writes nothing; the running host now holds tcode.db and tcode.lock with locks that Windows enforces, so the read failed there. The snapshot skips those files and the test compares the event log and the index through the store instead.
…tore files by name The store's lock file cannot be read while it is held on Windows, so the migration snapshot records store files by name instead of bytes. An in-process restart right after a host stops can find the data directory still locked: a spawn the stopped host started (a git status refresh) keeps a copy of the lock descriptor until the child execs. The app restarts as a new process, and the test now does the same.
Owner
Author
|
Two more CI failures after the previous push, both settled at step 1 (driver, not code):
|
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sessions.jsonand every<id>.jsonlmove into one Turso database,tcode.db, owned bySessionStore. Closes #520 and #523 (as re-scoped: event bodies go into the database too; the engine is Turso by the maintainer's decision). Supersedes draft #591.Behaviour
PRAGMA user_version = 1):projects(id, body)andsessions(id, body)hold each row's serde JSON unchanged;events(session_id, position, line)holds each raw log line including its delimiter, so CRLF, blank lines, invalid UTF-8 and a missing final newline survive andread_event_log(export) is byte-identical to the source.read_eventsstill parses tolerantly.synchronous = FULLand, on Apple,fullfsync = ON, both read back at open. The store writer runs on its own thread and commits its queued database writes as one transaction (≤ 256 writes / ~4 MiB); settings, secrets, caches andFlushare ordered boundaries. A failed commit is reported once per write with its existingRuntimeError, never replayed;Flushnow carries aResult, and the first failure makes every later flush fail, so a released log is kept in memory rather than evicted on a failed barrier. An external import, a Tcode-export import and a fork each commit events and metadata in one transaction.catch_unwind, a panic or an unconfirmed rollback puts the store into a shared failed state, further operations error, nothing is checkpointed, and the host shuts its providers down and reports the error.tcode.lockis taken withFile::try_lockfor the host's lifetime before any migration artifact is touched; Turso's own exclusive file lock is a second guard. A second host getsResourceBusynaming the data dir: headless exits 1, the desktop shows a critical prompt and exits 1. With the macOS relaunch marker present the open retries for up to 15 s; the marker is only consumed after ownership is acquired.open_atstays directory-only, so subcommands never open the database.tcode.db: buildtcode.db.migrating(schema,user_version0); import the index with today's tolerance (object or bare array,migrate_indexonce, duplicates fail the migration, a corrupt index is preserved as.corrupt-*); import every*.jsonlfound in the directory (orphans kept) in ~8 MiB transactions and verify each thread segment by segment against the file; read the metadata back field for field; setuser_version = 1last; checkpoint (result checked, WAL must be 0 bytes), drop every handle, reopen and re-check counts, fsync the file; write the archive list, fsync, rename totcode.db, fsync the directory; then move the sources intolegacy/(idempotent, never overwrites). Before the rename nothing is published and the next start discards the staging files; after it the next start only finishes the archival. Auser_versionof 0 or above 1, or an unopenable file, is refused with a message naming the file and thesqlite3CLI; the file is never deleted. A fresh install creates its database through the same staged path.(len, mtime). Nothing openssessions.jsonor<id>.jsonlany more except the migration. The startup whole-index rewrite is gone; reads are fallible internally (no protocol change).deliver_child_callbackread a child's transcript while its last appends were still queued; it now uses the cached fold or a store barrier.Evidence
tcode.db10.6 GB, WAL 0 after shutdown,sqlite3 PRAGMA integrity_check=ok; restart in 1 s with nothing migrated; threads render in the desktop app; a second headless host and a second desktop instance are refused with exit 1.tcode.dband sources intact, or a complete one with every source byte-identical in place or inlegacy/.turso = "=0.8.1"without default features (no mimalloc global allocator, no FTS). Clean release build +126 s (+33 %), desktop binary +12.5 MB, 66 new packages inCargo.lock. Windows targets needrc.exeat build time (present on the CI runner; a local cross-check from macOS needs a stub).Tests
Added at the store API: a migration fixture with legacy bare events, CRLF, blank, unparseable and non-UTF-8 lines, an unterminated last line, an empty log, an orphan log and removed meta fields; legacy bare-array index; unparseable index; duplicate ids; interrupted migration (torn staging, complete staging before the rename); interrupted archival; fresh install; refused
user_version0/2 and garbage files; exact bytes through clone and replace; a real second process refused and a relaunch waiting; two simultaneous first launches. Runtime: a host on legacy files migrates once and serves the same timelines across a restart; failed writes are reported and keep the unsaved history (fails on the old evict-on-any-barrier code). Two ignored SIGKILL harnesses stay as evidence tools.Changed: tests that read or wrote the files now use the store (fork compares
read_event_log; the search test replaces content through the store; the icon test reopens the store per restart as a process would). Removed: the icon test's failure injection via a directory atsessions.json.tmp(no database equivalent; the disk-full run covers a failed commit); two store tests folded into the migration fixture.Checks run
cargo fmt --all --check,cargo clippy --workspace --all-targets --locked -- -D warnings,cargo nextest run --workspace --locked(899 passed, 10 skipped),cargo machete, Web (wasm32) and iOS simulator checks;cargo xwin clippyforx86_64-pc-windows-msvcandcargo zigbuildforx86_64-unknown-linux-gnuon the services and runtime crates. Not run: Windows and Linux tests, Android. An intermittent nextest LEAK on unrelatedtcode-servicesunit tests reproduces onmain.PROTOCOL_VERSIONbump: a note was added under "Unreleased" above theconstant in
crates/protocol/src/lib.rs(the number itself changes onlywhen the release is cut — CONTRIBUTING.md, principle 9).
Migration dialog (second commit)
The migration is a separate cancellable step (
SessionStore::needs_migration/migrate(progress, cancel)) run under the ownership lock before the host starts;open()refuses an unmigrated directory. The desktop opens a window with a modal dialog first (gpui-base dialog and progress primitives as they are): title, one line of explanation, progress bar, phase and counts, and a Quit button; nothing else is reachable. Quit, ⌘Q or closing the window sets cancel, the migration stops at its next chunk or verification page, discards the staging files and the process exits 0; on success the kernel and shell start in the same process. A failure stays in the dialog and Quit exits 1. Headless prints progress at most once a second and treats SIGINT and SIGTERM as cancel (SIGTERM now gets the same graceful shutdown as SIGINT for the whole ofserve).No source is written, renamed or removed before the staged database is renamed to
tcode.db; cancel is never checked after the last pre-publication point, so publish → archive always completes. One fix on the way: the build used to rename an unparseablesessions.jsonto.corrupt-*; it is now left alone and archived unchanged.Looked at in the app on scratch data: light and dark, default and 360 pt width; Quit, ⌘Q and the close button mid-import and mid-verify left every source byte-identical and no
tcode.db; the next start completed; a forced failure exited 1. Known: at 360 pt the zh-CN description wraps CJK punctuation onto its own line (gpui's line breaker); the failure detail is the English error text.Tests: a cancelled migration (at four points) changes no source and the next one completes with the phases in order and done == total; headless SIGINT during migration exits 0 with the directory unchanged (real child process); existing migration tests now migrate before opening, as the app does. Workspace suite 901 passed.