Skip to content

perf(snapshot): add mutex lock, incremental add, and batched revert#8445

Merged
johnnyeric merged 1 commit into
mainfrom
johnnyeric/git-snapshot-optimizations
Apr 7, 2026
Merged

perf(snapshot): add mutex lock, incremental add, and batched revert#8445
johnnyeric merged 1 commit into
mainfrom
johnnyeric/git-snapshot-optimizations

Conversation

@johnnyeric

Copy link
Copy Markdown
Contributor

Port snapshot optimizations from upstream OpenCode to reduce git process spawning and prevent concurrent corruption.

Relates to #8379

Port snapshot optimizations from upstream OpenCode to reduce git process spawning and prevent concurrent corruption:

  • Per-gitdir mutex lock (upstream #17878, v1.3.0): serializes concurrent snapshot operations to prevent race conditions
  • Incremental git add (upstream #17878, v1.3.0): replaces naive git add . with diff-files + ls-files + size filtering, skipping files >2MB
  • Batched revert (upstream #20564, v1.3.14): groups up to 100 files per git checkout call with path-conflict detection, falling back to single-file revert on failure

Benchmarks (200 files):

  • revert: 2,851ms → 191ms (15x faster, 8x fewer git processes)
  • concurrent track: 2/5 succeed → 5/5 succeed (corruption fixed)

Context

Users reported high RAM usage and excessive git processes, especially on Windows with Agent Manager. A potential root cause is the snapshot module spawning many concurrent/redundant git processes during track, revert, and diff operations. OpenCode upstream has addressed these with Effect-TS based optimizations, this PR ports the key improvements as plain async/Promise code without adopting the Effect runtime until we are able merge newest upstream releases.

Implementation

Three optimizations ported from upstream, all contained in packages/opencode/src/snapshot/index.ts:

  • Per-gitdir mutex lock (upstream #17878, v1.3.0): A simple async lock (withLock) serializes concurrent snapshot operations per gitdir. Prevents race conditions where parallel track() calls corrupt the snapshot index. Without the lock, only 2/5 concurrent calls succeed; with it, all 5 return consistent results.
  • Incremental git add (upstream #17878, v1.3.0): Replaces naive git add . (which walks the entire worktree) with git diff-files + git ls-files --others to find only changed/new files, filters out files >2MB via concurrent stat, then runs git add --sparse. Reduces work on large repos.
  • Batched revert (upstream #20564, v1.3.14): Groups up to 100 files per git checkout call with path-conflict detection. Falls back to single-file revert on failure. Reduces 60+ sequential git process spawns to ~6.

Local Adhoc Benchmarks (200 files)

Operation Before After Improvement
revert (60 files) 2,851ms / 65 git processes 191ms / 8 git processes 15x faster, 8x fewer processes
5x concurrent track 2/5 succeed 5/5 succeed Corruption fixed

Screenshots

N/A -- backend optimization, no UI changes.

How to Test

  1. Run the snapshot test suite: cd packages/opencode && bun test
    test/snapshot/snapshot.test.ts test/kilocode/snapshot-cache.test.ts
  2. All 57 tests should pass (11 new tests covering concurrency, batching, and large file exclusion)
  3. For manual testing: use Agent Manager to edit multiple files in a session and monitor git process count / RAM usage --should see significantly fewer git processes and more stable memory
  4. PENDING: Windows testing

Get in Touch

@johnnyeric johnnyeric requested a review from imanolmzd-svg April 6, 2026 23:55
Comment thread packages/opencode/src/snapshot/index.ts Outdated
Comment thread packages/opencode/src/snapshot/index.ts
@kilo-code-bot

kilo-code-bot Bot commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 0

Fix these issues in Kilo Cloud

Issue Details (click to expand)

WARNING

File Line Issue
packages/opencode/src/snapshot/index.ts 427 Oversized tracked files are still staged because git add --sparse . updates tracked entries even when they are listed in info/exclude.
Other Observations (not in diff)

Previously reported non-ASCII batched revert issue is resolved via core.quotepath=false. Previously reported unborn-HEAD git reset -- <paths> issue is resolved by removing that fallback.

Files Reviewed (2 files)
  • packages/opencode/src/snapshot/index.ts - 1 issue
  • packages/opencode/test/snapshot/snapshot.test.ts

Reviewed by gpt-5.4-20260305 · 221,966 tokens

@johnnyeric johnnyeric force-pushed the johnnyeric/git-snapshot-optimizations branch from b450bb9 to 0c1dfa1 Compare April 7, 2026 10:04
Comment thread packages/opencode/src/snapshot/index.ts Outdated
@johnnyeric johnnyeric force-pushed the johnnyeric/git-snapshot-optimizations branch 2 times, most recently from be6aa4c to af3dea0 Compare April 7, 2026 10:27
Port snapshot optimizations from upstream OpenCode to reduce git process
spawning and prevent concurrent corruption:

- Per-gitdir mutex lock (upstream #17878, v1.3.0): serializes concurrent
  snapshot operations to prevent race conditions
- Incremental git add (upstream #17878, v1.3.0): replaces naive `git add .`
  with diff-files + ls-files + size filtering, skipping files >2MB
- Batched revert (upstream #20564, v1.3.14): groups up to 100 files per
  git checkout call with path-conflict detection, falling back to
  single-file revert on failure

Benchmarks (200 files):
- revert: 2,851ms → 191ms (15x faster, 8x fewer git processes)
- concurrent track: 2/5 succeed → 5/5 succeed (corruption fixed)
@johnnyeric johnnyeric force-pushed the johnnyeric/git-snapshot-optimizations branch from af3dea0 to a010010 Compare April 7, 2026 10:33
@johnnyeric johnnyeric merged commit afe950d into main Apr 7, 2026
13 checks passed
@johnnyeric johnnyeric deleted the johnnyeric/git-snapshot-optimizations branch April 7, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants