Skip to content

feat(overleaf): coordinator local half — apply, recover, adopt, kept copies (step 3b) - #630

Merged
yaacovcorcos merged 16 commits into
mainfrom
claude/overleaf-coordinator-local-20261022
Oct 23, 2026
Merged

yaacovcorcos merged 16 commits into
mainfrom
claude/overleaf-coordinator-local-20261022

Conversation

@yaacovcorcos

@yaacovcorcos yaacovcorcos commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What this adds

Step 3b of the Overleaf plan (#535; design in docs/internals/overleaf/step-3b-design.md there): the coordinator's local half. OverleafCoordinator prepares a review, applies a confirmed one to the project folder, recovers an interrupted apply, and adopts the result into the connection record. It also lists the copies Scient kept of replaced files and restores one on request.

It never publishes. Step 4 continues from an adopted result, and step 5 connects it to the app behind sign-in. Until then nothing calls it, so users see no change.

How it works

Locks, in order. The connection's operation reservation comes first: it's held in memory, and a second operation is refused (OverleafOperationActiveError), not queued. Then come the process's writable lease (admitWritable), the checkout barrier's application section (#626), the record's compare-and-set, and the file layer's own locks. Overleaf is fetched before the section is taken.

Confirm:

  • It takes the review once.
  • Inside the section, it checks the checkout, the folder and the retention folder are the ones recorded, captures the folder again, and rebuilds the review with the same selections and choices. A different digest means the review is stale: the fresh review is offered and nothing is written.
  • Capture reads the base's paths, the managed files and every path in a forced conflict, so a file an interrupted apply left outside both the base and the result is still reviewed.
  • Before anything is written, it checks there's room in the record for every copy the apply could keep, and for every group an interruption could leave as a forced conflict (2,000 paths each at most).
  • Then, before the applier runs, it records, in this order: the receipt, a local snapshot, the pinned intent, and the apply link.

Adoption is one replacement of the connection record, so it happens exactly once. It sets:

  • the new base, written from the applier's per-path base on the fetched commit, with the previous base kept among prior bases;
  • the managed files: the reviewed result's paths;
  • the local-only files: exactly the set the review counted;
  • forced conflicts (interrupted groups) and conflict-marker paths;
  • kept copies (see below);
  • the completed-sync count, which advances only on a complete apply;
  • unknown publication attempts, marked superseded;
  • the apply link, cleared.

Recovery resumes the applier by the linked ID and adopts its result once. A link whose apply never recorded its plan is cleared as a replan.

Kept copies:

  • Records. There's one record per file an apply replaced. It names every version the applier kept, its revision, a size-and-change-time stamp, and whether it was Scient conflict material. The stamp is taken around a fresh read of the kept bytes at adoption, from the file the path still names. A copy that no longer holds what the applier kept (a late writer changed it) is recorded as changed.
  • Release. A record with any extra version is never released automatically. Every sync and restore rechecks older copies by their stamp, and one that has changed or gone missing is never released.
  • Listing compares each copy by its stamp and reads again only changed or extra versions. It reads at most one inventory's total (250 MiB). A file past that budget, or larger than a restore reads (kept copy or folder file), is listed as unread, never as gone.
  • Restore. Restore puts a chosen version back as a one-file apply, only if the version and the file it replaces still match what the caller saw. It reads the version and the folder's file once each, never past 50 MiB, and never takes a file too large to read for an absent one. It keeps the file it replaces, never moves the base, and keeps local-only files local-only. A restore another writer interferes with is kept as a conflict for the next review. Restored conflict material is tracked again. The restored file is then a local change for the next review.
  • Room. There's room for every record, for a file a restore brings back, for the conflict it may leave and for conflict material it tracks again, before anything is written.

Bulk changes. The coordinator registers a settler with the barrier. Before a pull, branch switch or checkpoint restore of a checkout with an unfinished apply:

  • it recovers the apply inline when the connection is free, and refuses the bulk change otherwise;
  • it also refuses while temporary copies the applier made during a restructuring still sit in the folder, including copies the inline recovery has just found;
  • it refuses whenever a connection record can't be read.

Record changes

Schema version 1 is unchanged, because nothing is stored in production yet.

  • attentionPaths becomes forcedConflicts, markerPaths and stagingPaths.
  • The apply link gains remoteCommit, localOnly and priorMarkers, so recovery can adopt without the review.
  • Kept-copy records gain kept, versions, stamp and markers.
  • A record holds up to 4,000 forced conflicts (was 2,000): one sync changes at most the 2,000 files it captured and the 2,000 of its result.
  • The applier reports each kept copy's path, recovers an apply that never recorded its plan as a replan, and exposes the groups an interruption could leave (interruptibleGroups, shared with its own planning).

Tests

OverleafCoordinator.test.ts has 35 tests, using real folders, a real private Git repository and the real native applier, against a scripted Overleaf.

  • Applying: a clean apply adopted once, duplicate and reused request IDs, a stale review, the second-operation refusal, refusal without the writable lease, and refusal before writing when an interrupted group couldn't be recorded.
  • Crashes and interruptions (thrown hooks between the coordinator's durable steps): a crash after the applier ran and one just after linking, then an interrupted group becoming a forced conflict.
  • Bulk changes: a bulk change waiting for an apply, inline recovery before one, and refusal with copies in the folder (also ones the recovery finds) or a damaged record.
  • Kept copies:
    • listing and restore, with a stale restore;
    • room, managed-file and conflict-material limits;
    • every kept version, including extra versions behind a damaged step record;
    • rechecks of older copies, a copy changed between the applier's hash and adoption, and a copy whose path was swapped while it was read;
    • kept copies and folder files past the read limits, listed as unread and never restored over;
    • a path an interrupted group left outside the base and the result, read by the next review;
    • restored conflict material, local-only restores, deleted files sent again, and an interrupted restore kept as a conflict.
  • Mutation checks: each fix was removed in turn, and its test failed.

vp test run src/scient/overleaf src/scient/workspace: 562 tests pass. Typecheck, lint and knip are clean. These need macOS native publication and are skipped on Linux CI.

Review

Codex (gpt-6.1-sol, high reasoning) reviewed the 3b branch in eight rounds. Rounds 1–7 found the problems below, all fixed, each with a test that fails without its fix. Round 8 found nothing: merge-ready (89 focused tests, 30 independent probes and three real crash-and-recover runs passed).

Round Findings
1 Kept versions of one file overwrote each other; the bulk-change settler failed open; restore had no room check; local-only adoption disagreed with the review; a restored deleted file wasn't sent again; restores reported the wrong generation.
2 "Three versions per path" wasn't a real bound; restore could pass the managed-file limit; older copies were never rechecked; restore lost conflict-marker provenance; restore promoted local-only files.
3 Interrupted adoption could pass the managed-file limit; prepare skipped the process lease; non-main versions weren't rechecked; a damaged step record hid versions; hashing had no overall budget.
4 A kept copy's stamp was taken after the applier's hash, so a same-size rewrite in between was never noticed; a path an interrupted group left outside the base and the result was refused by the next review; the settler didn't recheck for copies its own recovery found; restore and listing reads weren't bounded.
5 One interrupted group could exceed the record's 2,000 paths, so adoption could never be written; a kept file swapped at its path just after opening still read as unchanged; the folder's file skipped the per-file read limit.
6 Restoring kept conflict material could take the record's marker list past its limit after writing the file.
7 An interrupted restore left no conflict to review, so the next review called the folder up to date.

Since round 4 the branch has also taken main (including the upstream alignment #614). Its only clashes were re-indentation, and it now follows main's new lint rules.

Follows #624 and #626.

🤖 Generated with Claude Code

yaacovcorcos and others added 10 commits October 22, 2026 21:04
…s-20261022' and 'origin/claude/overleaf-checkout-barrier-20261022' into claude/overleaf-coordinator-local-20261022
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s-20261022' and 'origin/claude/overleaf-checkout-barrier-20261022' into claude/overleaf-coordinator-local-20261022
… test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s-20261022' and 'origin/claude/overleaf-checkout-barrier-20261022' into claude/overleaf-coordinator-local-20261022
… room first

Codex round 1 on step 3b:
- Kept copies are identified by their place in the retention folder, so every
  version a step kept of one path is recorded and restorable.
- The bulk-change settler fails closed: an unreadable connection list or
  record, or a folder it cannot look up, refuses the bulk change.
- Restore checks room for what it will keep before linking; confirm counts up
  to three kept versions per changed path.
- The apply link carries the local-only set the review counted, and adoption
  uses exactly that.
- A restored file takes part in the sync again (managed, not local-only), so
  the next review sends it; restores report an unchanged base generation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ked, provenance

Codex round 2 on step 3b:
- One kept record per (apply, path); every version a step kept (main, staged,
  provisional, a retried step's) is listed and restorable from disk. Retries
  can no longer outgrow the room check, which is now exact per changed path.
- Every sync and restore rechecks older releasable copies and records them
  changed or ambiguous, so a late write is never released.
- Each record keeps whether its bytes were unresolved Scient conflict
  material; restoring that version tracks the path's markers again.
- Restore keeps a local-only file local-only, and checks the managed-path
  limit before writing. The limit error says which limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…admission

Codex round 3 on step 3b:
- Each kept record names every version the applier reported for its path
  (bounded; beyond it the record is ambiguous), so listing and restore no
  longer depend on the applier's step records on disk. A record with any
  further version is never releasable.
- Rechecks compare a size-and-change-time stamp recorded when kept, with no
  hashing; listing re-reads only changed or extra versions and each current
  file once.
- Managed paths after an apply are the reviewed result's paths, bounded by
  the review's file limit; paths an interrupted unit kept stay in the base.
- Every operation takes the store's writable admission (process lease)
  before touching the private repository; the store exposes admitWritable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL labels Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.6 KiB 5.6 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 4.3 KiB 4.3 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.3 KiB 1.3 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 26.2 KiB 26.2 KiB 0 B (0.0%) 29.3 KiB ✅
Codex Live turn messages 2 2 0 (0.0%) 8 ✅
Claude Total thread wire 5.6 KiB 5.6 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 4.3 KiB 4.3 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.3 KiB 1.3 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 26.6 KiB 26.6 KiB 0 B (0.0%) 29.3 KiB ✅
Claude Live turn messages 2 2 0 (0.0%) 8 ✅

Baseline: 3ca9472 · PR result: b4e2569 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Full compact HTTP snapshot with live cursor.

  • Codex decoded thread snapshot: 133.1 KiB
  • Claude decoded thread snapshot: 133.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

yaacovcorcos and others added 6 commits October 23, 2026 11:08
The alignment brought `prefer-effect-fn` and `no-real-time-wait-in-tests`.
`make` and `keptCopies` are now `Effect.fn` functions (traced spans, no
behaviour change). The bulk-change test no longer sleeps 100 ms for the
competing bulk change to reach the barrier: it waits until the checkout's
epoch moves, which the bulk change does as it enters, before it waits for
the apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ptured, bounded reads

Codex round 4 on #630:
- A kept copy's stamp was taken after the applier hashed it, so a late
  writer's same-size rewrite in between looked like the kept bytes forever.
  Adoption now reads the copy afresh (stat before and after a bounded read):
  only bytes that still match what the applier kept get a stamp; anything
  else is recorded changed (ambiguous when gone) and is never released.
- A path an interrupted group left in neither the base nor the result (an
  undone rename's new name) fell out of capture, and the next review refused
  it as unreviewed. Capture now reads every forced-conflict path too; nothing
  stored grows.
- The bulk-change settler checked for structural copies only before its
  inline recovery; one the recovery found was then erased by the bulk change.
  It now re-reads the adopted record and refuses while copies remain.
- Restore reads the kept copy once, never past 50 MiB, and restores exactly
  the bytes it checked. Listing reads within one inventory's total; files past
  it or past the restore limit are listed `unread`, not gone.

The bulk-change ordering test checks at adoption that the bulk change had
entered the barrier, instead of polling for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r-file limit

Codex round 5 on #630:
- An interrupted group is stored whole as a forced conflict, and one group
  can exceed the record's 2,000 paths (a 2,000-file folder replaced by a
  file). Adoption then failed forever, leaving the apply linked. The
  applier's unit computation is now shared (`interruptibleGroups`), and
  confirm checks every group an interruption could leave before linking or
  writing anything. The record holds up to 4,000 groups: one sync changes
  at most the 2,000 files it captured and the 2,000 of its result.
- A kept file swapped at its path right after being opened still read as
  the unchanged copy. A read now also requires the path to name the file
  the descriptor read.
- Listing and restore read the folder's file within the same per-file limit
  as kept copies; a restore never takes a file too large to read for an
  absent one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex round 6 on #630: restoring a kept copy that held conflict material
tracks its path again, and with 2,000 marker paths already recorded that
took the record past its limit only after the file was written, leaving
the restore linked for good. Restore now checks room for the marker paths
it would track, with its other room checks, before reading, linking or
writing anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rced conflict

Codex round 7 on #630: restore adoption kept the record's forced conflicts
but dropped the group the applier reported as interrupted, so a restore
another writer interfered with left nothing to review and the next review
called the folder up to date. Restore adoption now keeps the recorded
groups and adds the interrupted ones, without duplicates or truncation,
and a restore checks room for the one group it may leave before writing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yaacovcorcos
yaacovcorcos marked this pull request as ready for review October 23, 2026 11:19
@yaacovcorcos
yaacovcorcos added this pull request to the merge queue Oct 23, 2026
Merged via the queue into main with commit 52eecdd Oct 23, 2026
43 checks passed
@yaacovcorcos
yaacovcorcos deleted the claude/overleaf-coordinator-local-20261022 branch October 23, 2026 11:40
yaacovcorcos added a commit that referenced this pull request Oct 23, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant