Skip to content

perf(git): index repeated tracked pathspec lookups - #35831

Closed
OrcaWin wants to merge 1 commit into
mainfrom
OrcaWin/git-tracked-partition-index
Closed

OrcaWin wants to merge 1 commit into
mainfrom
OrcaWin/git-tracked-partition-index

Conversation

@OrcaWin

@OrcaWin OrcaWin commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator
Files Added Deleted Net
Test 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​68 0 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​68
Prod 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​58 $\color{#cf222e}{\Huge{\mathbf{−}}}$​8 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​50

Bulk discard repeatedly searches the same tracked-path list while deciding which selections go to git restore and which go to safe untracked cleanup. With 1,000 individually selected tracked files, that partition performs 500,500 comparisons before the native write. This change reuses the existing operation-local normalized cache and switches repeated expensive scans to a sorted index.

The matcher stays lazy for small selections and cheap first-descendant matches. For multiple requests and at least two tracked entries, it budgets N × ceil(log2(N)) scan comparisons. A successful match returns immediately. Only a nonmatch with both remaining evidence and a future request builds the index: it fills the existing cache, sorts that same array once, and uses separate lower bounds for exact paths and slash-delimited descendants. A completely exhausted miss and the last request do not trigger construction. There is no second normalized array, Set, persistent cache, subprocess or wire change.

Measured work

These are deterministic counts from executed source, not timing or heap measurements. The production-shaped mixed case contains 700 selected tracked entries and 300 untracked selections, matching the existing selected ls-files output.

Workload Before After
1,000 exact tracked selections 500,500 scan comparisons 10,000 scans + one 1,000-string sort + 8,580 binary probes
700 tracked + 300 untracked selections 455,350 scans 7,000 scans + one 700-string sort + 10,960 probes
One directory with 1,000 descendants 1 scan 1 scan, no sort
Missing path then early directory match 1,001 scans 1,001 scans, no sort
1,000 repeated first-descendant matches 1,000 scans 1,000 scans, no sort
First-child matches then final-only miss 11,000 scans 11,000 scans, no sort

Sort population is measured; the engine's sorting comparisons, scratch allocations, CPU, latency and retained heap are unmeasured. An earlier pure mixed benchmark included unselected tracked evidence; it remains explicitly labeled API-only and is excluded from the production mixed count above.

Correctness and validation

  • The existing normalizer, public signatures, literal/case/prefix semantics, original output spellings, duplicate selections and relative action order are preserved. Caller arrays are never sorted.
  • Independent execution of the captured public exports agrees on 10,026 partition and 2,001 scalar comparisons, including frozen inputs, lexical neighbors, backslashes, separators, Unicode, drive/UNC-looking strings and glob-like literals.
  • 70 focused tests across seven files pass on each of Git 2.25.5 and 2.55.0, covering public bulk discard, symlinks, index safety, banner handling and watcher-suspect writes. The final regression suite fails two cost assertions on the exact parent and passes its other 28 controls.
  • Actual uninstrumented public bulk-discard calls cover ten native cases per source variant (two Git versions × five workloads). Commands, stdin, selected paths, resulting status, tracked bytes, untracked cleanup and outside-worktree refusal agree exactly. All 30 loaded parent/final source records were checked against their pinned trees.
  • ORCA_BACKGROUND_LAUNCH=1 pnpm check:changed --base=0e9ba2167b8cd16bffddce50c908bf0400da10e2 --maxWorkers=2: all 20 checks pass in 1,158.5 seconds; 1,228 affected test files and 14,166 tests pass (7 expected failures and 96 skips). Actual worker temp paths used the short task-owned cache root. All 400 observed process identities exited and the task temp root is empty.

This is synchronous, operation-local host work. Existing Git routing, chunked command admission, invalidation and cancellation ownership remain unchanged. Desktop, web/mobile and folder workspaces share the existing API; there is no new git-worktree assumption. Native proof ran on Linux. Live macOS, Windows, WSL, SSH and mobile execution, rendered UI, whole-app responsiveness and heap behavior were not measured.

Integration before merge

Coordinate with #27757 (POSIX/Windows tracked-path normalization) and its related WSL input work #28862. This draft deliberately preserves its parent's normalization policy; #27757 changes the same module/public API and the bulk-discard caller. Compose or restack the changes and retain both sets of controls before merge. This PR does not implement or supersede that path-policy correction.

Source is pinned to parent 0e9ba2167b8cd16bffddce50c908bf0400da10e2, candidate tree fd71c534d06cc1581aa5dad5c1b3390aedcb756e. Detailed raw evidence is retained locally in tracked-partition-index-actual-v2, tracked-partition-index-independent-git-exec and tracked-partition-index-gate packets. The two source/test files are the complete product delta.

@OrcaWin

OrcaWin commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: the saving is a few milliseconds of in-memory compares, only on discards of hundreds to thousands of files, next to git restore/clean subprocesses that cost far more, and the PR measured no latency. It adds stateful budget and index switching to the tracked/untracked partition, where a mistake would delete files instead of restoring them.

@OrcaWin OrcaWin closed this Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant