Indexing consolidation: unify vector planners, clarify reindex mechanisms, micro-batching

[phernandez]

From the 2026-07 architecture review (basic-memory-cloud docs/architecture-review-2026-07.md, diagram 4). Lower priority — comprehension debt plus modest wins, not a live bottleneck.

Important constraint first: both runtimes run FastEmbed (CPU); semantic_embedding_sync_batch_size = 2 is the deliberately benchmarked local default and cloud overrides to 64 via env. Do not change flush behavior without re-running those benchmarks.

• Three overlapping vector planners: indexing/embedding_index_planning.py, indexing/vector_sync_planning.py, plus the batch path — unify to one planner with single + batch entry points
• Three "reindex" mechanisms with easily-confused semantics: reindex_all_tenants (FTS + vectors, cloud), reindex_search -> search_service.reindex_all() (FTS-only, sequential per entity), reindex_vectors — operator guidance documented in basic-memory-cloud docs/architecture-review-2026-07.md
• Incremental path enqueues one INDEX_EMBEDDINGS PGQ job per entity (file_followup_jobs.py) — coalesced into bounded embedding batches in basicmachines-co/basic-memory-cloud#1572
• Vector prepare opens one write transaction per entity (search_repository_base.py:1421-1558) — batch
• Relation resolution opens a new session per relation and per affected entity (indexing/relation_resolution.py:150-205) — batch into set-based updates + one multi-entity reindex

[claude]

Claude finished @phernandez's task in 1m 56s —— View job

---

Triage complete

• Read issue and gather context
• Classify type, priority, component, complexity
• Check for duplicate issues
• Apply labels
• Post triage summary comment

Type: Enhancement (internal refactor/tech-debt, no user-facing behavior change intended)

Priority: Low — the issue itself says so explicitly ("comprehension debt plus modest wins, not a live bottleneck"), consistent with it coming out of the 2026-07 architecture review rather than an incident.

Component: Spans database/indexing (this repo) and cloud (the cross-tenant reindex path). Confirmed:

• src/basic_memory/indexing/embedding_index_planning.py, vector_sync_planning.py, relation_resolution.py, and src/basic_memory/repository/search_repository_base.py (2,382 lines — the :1421-1558 range is a plausible vector-prepare section) all exist in this repo (basic-memory).
• reindex_all_tenants and file_followup_jobs.py do not exist anywhere in this repo — those live in basic-memory-cloud. So this ticket's scope crosses two repos; the "which reindex mechanism to use when" consolidation in particular can't be fully resolved from basic-memory alone.
• Applied the cloud label in addition to enhancement to reflect that split. (Note: the priority/component/complexity label taxonomy described in the triage prompt — e.g. low, complex, database — isn't provisioned in this repo's label set, so only enhancement and cloud were applied; everything else that exists is bug/documentation/duplicate/question.)

Complexity: Complex. Five distinct sub-items, each touching hot-path indexing/search code, with an explicit hard constraint already called out in the issue: semantic_embedding_sync_batch_size (local=2, cloud=64 via env) was deliberately benchmarked, and flush-behavior changes must not proceed without re-running those benchmarks. Recommend treating this as a tracking issue split into independently landable PRs (planner unification, reindex-mechanism docs/consolidation, job coalescing, vector-prepare batching, relation-resolution batching) rather than one large change, since they touch different files/services and carry different risk levels (the relation-resolution and vector-prepare batching changes are the riskiest, as they touch transaction/session boundaries).

Duplicates: None found. Related but distinct: #1108 ("Split core god files") comes from the same architecture review (diagram 2 vs. this issue's diagram 4) and also touches search_repository_base.py, but it's about file-splitting for readability, not planner/reindex consolidation — no overlap in scope to flag as duplicate.

No reproduction steps or clarification needed — this is a well-scoped internal refactor proposal with concrete file references, not a bug report.

[phernandez]

Completed. The final incremental path now coalesces webhook-local file follow-ups into bounded embedding batches in basicmachines-co/basic-memory-cloud#1572. The reindex semantics and operator guidance are documented in the merged Cloud architecture review, including the important warning that Cloud reindex_search clears vector tables but repopulates FTS only.