[nateEc]

Every pull currently rereads and tokenizes every knowledge file, including the unchanged-revision path. This caches per-file derivations while recomputing selection, source precedence, votes, confidence and document frequencies on every build. Deletions and inactive namespaces disappear from both the index and its cache.

Fingerprints include device/inode, size and nanosecond mtime/ctime. Metadata-only changes compare content hashes before reusing tokens. Schema, derivation, Node or ICU changes invalidate reuse. Selected entries are referenced instead of storing their tokens twice. Publication remains atomic, with reused/rebuilt/dropped counts in the debug log.

Verification

• Build, typecheck and lint passed; 159 focused search tests passed.
• Real CLI: npx vitest run --config vitest.e2e.config.ts src/__tests__/e2e/incremental-index.test.ts passed. Three real pulls in an isolated HOME with a local Git remote built the index, reused unchanged sources (0 rebuilt), then indexed an upstream doc edit.
• Full unit run: 12,175 passed, 36 skipped, 6 failed, plus one worker RPC timeout under parallel load. The three workflow-workspace timeouts passed when the full 122-test file was rerun with two workers; the zsh startup assertion passed alone. The two remaining submodule-checkout failures also reproduce on untouched origin/main (97f1e7b): git config --get hook.teamai-post-checkout.command is missing in this environment.

Single local-disk samples with ~2 KB Markdown files while the unit suite was running:

Files	Full build	Unchanged	One edited	100 edited
1,000	3,813 ms	224 ms	108 ms	227 ms
5,000	4,530 ms	331 ms	327 ms	251 ms

NFS was unavailable. Enumeration/stat, index loading, frequency aggregation and atomic serialization remain O(N); tests assert that unchanged sources are neither read nor retokenized. Coverage includes same-size edits with restored mtime, deletion, category/selection changes, votes, and equivalence with a full build.

Fixes #1239

Follow-up verification (2026-10-25)

• Fixed user-scope cache misses caused by the learning mirror overwriting identical files. Root and namespaced mirror copies now retain their metadata only when both bytes and permissions are unchanged. Upstream executable-bit changes still propagate to namespace attachments.

• Added a timestamp regression (red before the fix) and a real built-CLI read probe: three unchanged user-scope pulls produce zero learning reads from entryFromMdFile. Editing the learning then triggers reads, proving the probe is active, and updates the index.

• Learning-mirror and incremental-index unit files: 14 passed. Built-CLI incremental-index E2E: 1 passed. Build, type checking and lint passed.

• Cost note: the incremental cache stores reusable derived entries and makes the index larger; the prior review measured 2.19 MB to 7.41 MB at 5,000 user learnings. Mirroring still compares source/destination bytes; the zero-read guarantee applies to the index build, not every operation in pull.

• Follow-up review: reproduced the executable attachment permission issue with a failing regression and fixed it; the final built-CLI read probe still passes after that correction.

Updated head: 8d6488299cf13c3e51b1ae384ea6d9824648da13. Rebased onto main 6ebb31ce.

[github-actions]

No confident findings in the specified PR diff.

The PR description satisfies the testing requirement: it records a passing real-CLI E2E test covering initial indexing, unchanged-source reuse, and an upstream document edit. One representative run is sufficient.

Review was read-only; I did not run, build, or install PR code.

[SaulMoro]

This was generated by AI during triage.

@nateEc Thanks, this is a solid change. Project scope does what #1239 asks. One acceptance item is still open in user scope.

We built this head and its base 97f1e7b4 and ran teamai pull with a read probe on the index build. We ran add, edit, rename, delete and vote steps. At every step the index equals a full rebuild. An index written by either build is read correctly by the other.

[P1 blocking] In user scope, an unchanged pull still reads every learning. The triage on #1239 put the criterion as "on an unchanged pull, the index build reads and tokenizes no source file". It also named the user-scope mirror as the case that defeats a timestamp fingerprint.

• Scenario: user scope, 1,000 learnings, no change. Three pulls in a row each make 2,000 readFile calls from the index build on ~/.teamai/learnings/*.md. Project scope makes 0.
• Cause: src/utils/learnings-mirror.ts:111 and :115 copy every mirrored file with overwrite: true on each pull. That resets their timestamps, so the fingerprint misses (search-index.ts:521). The file is read, the hash matches, and only tokenizing is skipped.
• At 5,000 learnings an unchanged user-scope pull takes 336 ms, against 92 ms in project scope. The base takes 1476 ms.
• Two ways out. One is to make the mirror skip copies that are already identical and keep their timestamps. A scratch version of that took the reads from 2,000 to 0 and kept the index equal to a full rebuild across an edit and a delete. The other is to change Fixes #1239 to Refs #1239 and state the user-scope limit. Either way, the user-scope e2e should assert a read count, not only the reuse counts in the log.

Not blocking:

• [P2 non-blocking] The index file grows. With 5,000 learnings it goes from 2.90 to 5.18 MB in project scope, and from 2.19 to 7.41 MB in user scope. Every pull parses it. In user scope the growth comes from storing the shadowed copies from the inherited root as full entries. Please state the size cost in the body, and consider leaving unselected copies out of the cache.
• [P2 non-blocking] FILE_DERIVATION changes only with the schema, Node or ICU. A future tokenizer change keeps stale tokens unless someone bumps 1: by hand, and the triage noted derivation has changed before without a bump. Tying it to a version the build already has would close that.
• [P2 non-blocking] Two changes #1239 did not ask for. The Search index build took … consider incremental updates warning is removed. Carried-over (unreadable) entries now get current votes, and no test covers that.

[nateEc]

Addressed the review follow-up at 11868a83692a94b31d289f9e4ec332cb91c5c5d4 on current main.

• Fixed user-scope cache misses caused by the learning mirror overwriting identical files. Root and namespaced mirror copies now retain their metadata when bytes are unchanged.
• Added a timestamp regression (red before the fix) and a real built-CLI read probe: three unchanged user-scope pulls produce zero learning reads from entryFromMdFile. Editing the learning then triggers reads, proving the probe is active, and updates the index.
• Learning-mirror and incremental-index unit files: 13 passed. Built-CLI incremental-index E2E: 1 passed. Build, type checking and lint passed.
• Cost note: the incremental cache stores reusable derived entries and makes the index larger; the prior review measured 2.19 MB to 7.41 MB at 5,000 user learnings. Mirroring still compares source/destination bytes; the zero-read guarantee applies to the index build, not every operation in pull.

The PR description contains the updated verification record. Ready for another review.

[github-actions]

• [P2 non-blocking] Preserve permission changes when mirroring identical bytes (src/utils/learnings-mirror.ts:25). Namespaces mirror non-Markdown assets too. If an upstream commit makes alpha/helper.sh executable without changing its contents, this early return leaves the mirrored script non-executable; invoking it fails with “Permission denied.” Compare permissions before skipping, or restrict this optimization to Markdown files.

The PR description satisfies the testing requirement: it records a passing real-CLI E2E run and follow-up verification of user-scope cache reuse. One representative run is sufficient.

Review was read-only; I did not run, build, or install PR code.

[nateEc]

Addressed the review follow-up at 8d6488299cf13c3e51b1ae384ea6d9824648da13 on current main.

• Fixed user-scope cache misses caused by the learning mirror overwriting identical files. Root and namespaced mirror copies now retain their metadata only when both bytes and permissions are unchanged. Upstream executable-bit changes still propagate to namespace attachments.

• Added a timestamp regression (red before the fix) and a real built-CLI read probe: three unchanged user-scope pulls produce zero learning reads from entryFromMdFile. Editing the learning then triggers reads, proving the probe is active, and updates the index.

• Learning-mirror and incremental-index unit files: 14 passed. Built-CLI incremental-index E2E: 1 passed. Build, type checking and lint passed.

• Cost note: the incremental cache stores reusable derived entries and makes the index larger; the prior review measured 2.19 MB to 7.41 MB at 5,000 user learnings. Mirroring still compares source/destination bytes; the zero-read guarantee applies to the index build, not every operation in pull.

• Follow-up review: reproduced the executable attachment permission issue with a failing regression and fixed it; the final built-CLI read probe still passes after that correction.

The PR description contains the updated verification record. Ready for another review.

[github-actions]

No new confident findings in the specified PR diff.

• Resolved: The earlier permission-change finding is fixed: mirroring skips copying only when both bytes and permissions match, with an executable-bit regression test added.
• Testing: The PR description satisfies the requirement with a passing real-CLI E2E record and user-scope cache-reuse verification. One representative run is sufficient.
• Scope: Review was read-only; I did not run, build, or install PR code.

[SaulMoro]

This was generated by AI during triage.

@nateEc Thanks, approving at 8d64882. The blocking finding is fixed: the mirror no longer rewrites identical learnings, so their fingerprints hold. With 1,000 user-scope learnings, an unchanged pull now makes 0 reads of learnings from the index build, against 3,000 per pull on the base. Stale entries still rebuild with the built CLI in user scope: a same-size edit arriving through git with its old mtime restored, and an in-place same-size write keeping the inode, are both picked up, because ctime changes. recall finds the new tokens and no longer finds the old one. The PR's unit and e2e tests pass, also with a symlinked TMPDIR, and 4 unit tests and the e2e fail on the merge-base. Build, tsc and lint pass.

• [P2 non-blocking] A change to how entries are derived serves stale entries unless someone bumps FILE_DERIVATION by hand. With a build patched to change titleFromFilename and no bump, the next pull logged 3 reused, 0 rebuilt, the index kept the old title, and recall missed the file. The base rebuilt it. That makes the note in tokenizer.ts and the body's "derivation … changes invalidate reuse" misleading. Including the CLI version (getCurrentVersion()) in FILE_DERIVATION would reparse once per release and fail safe.
• [P2 non-blocking] Carried-over votes have no test.
• [P3 nit] The unchanged check trusts stat alone, so a same-size rewrite within one timestamp tick, or on NFS, could be missed. The user-scope index is about 2.4x larger (shadowed copies stored in full), which costs about 7 ms of parsing per recall at 5,000 learnings. The "Search index build took" warning is removed without a mention in the body.

The CI review reports no new findings. We agree that nothing blocks, but it ran no code and did not see the derivation case above. All CI jobs pass.