feat(core): add pluggable semantic vector indexes - #1141
Conversation
Signed-off-by: phernandez <paul@basicmachines.co>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0470b7d9db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
Resolved the failed semantic diagnostic job in signed commit |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 367a2f782a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d780c16f28
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8fa2182b0c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29b738901a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a9dda503c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fdc75d7f6a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3cf2133d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c5e3a0c0b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67f27ddc9e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d9aa17f768
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cbf8e50b3b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58d434f850
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 459ee6fc86
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b6367daec0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93715b2d91
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37074e361c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Why
Milvus support should not require moving Basic Memory's full-text search, embedding generation,
or SQL metadata into a vendor-specific repository. The contributor work in #1041 demonstrated the
need for a clean extension boundary, while also highlighting that an in-tree Milvus dependency
would create Windows compatibility and long-term ownership costs.
This PR establishes that boundary in core so Milvus can be delivered and maintained as an external
package. It also makes the vector lifecycle explicit enough to support future multimodal embedding
providers without coupling those providers to vector storage.
What Changed
SemanticVectorIndexprotocol for vector initialization, persistence, deletion,reconciliation, and nearest-neighbour lookup.
pgvectorandsqlite-vecimplementations behind the same contract.basic_memory.semantic_vector_indexes, allowing anexternal package to provide
semantic_vector_index = "milvus".migration and backend-neutral project embedding status.
adapter boundary.
Implementation Details
Core continues to own text chunking, embedding generation, and the authoritative SQL manifest.
Vector adapters receive already-computed vectors identified by stable
(entity_id, chunk_key)keys and own only vector storage and similarity search.
Writes first persist manifest rows as
pending. Every adapter write carries the chunk'ssource_hash: built-in adapters lock and verify that generation in the same transaction as theirvector write, while external adapters retain the matching manifest lock across adapter I/O and the
readytransition (FOR UPDATEon Postgres and a conditional write lock on SQLite). Searchresults are hydrated through current, ready manifest rows for the active project, embedding
identity, and source generation, so stale or cross-project external matches fail closed.
Deletes follow the same generation boundary. Cleanup stages the exact row, source hash, and owning
adapter, then rechecks that the manifest is still
pendingbefore deleting. External adapters rununder that manifest lock through the remote delete and exact SQL manifest removal; built-in
pgvector and sqlite-vec remove the matching vector and manifest row atomically in one database
transaction. A newer overlapping sync therefore wins without its vector or manifest being removed
by an older stale-row or whole-entity cleanup.
Project-wide cleanup uses the same serialization boundary as external writes. PostgreSQL locks the
project row before either operation, while SQLite holds its database write lock from ownership
discovery through adapter deletion and manifest removal. Full reindex and project cleanup therefore
cannot discard a manifest while a concurrent watcher publishes a new remote vector.
External reconciliation uses that boundary too: it holds the project lock from the ready-manifest
snapshot through
delete_orphans()and commit. A watcher therefore cannot publish a valid vectorafter the snapshot and have reconciliation delete it as an apparent orphan.
Project embedding status keeps that backend boundary explicit. External adapters remain
manifest-only because core cannot inspect their storage, while built-in pgvector and sqlite-vec
status requires the physical table and counts only current ready manifest rows with a matching
physical vector for the same source hash. A missing table, missing row, or unavailable sqlite-vec
runtime therefore recommends a rebuild instead of reporting a false healthy index.
Entity and project deletion now retain that serialization boundary through adapter cleanup and the
caller-owned SQL delete transaction. PostgreSQL records a durable
pendingretry marker beforeexternal deletion, while external prepare paths re-read and re-plan under the project lock so a
stale pre-lock snapshot cannot recreate a deleted entity's manifest. Explicit removal, config
reconciliation, and portable hard-delete paths all either clean external vectors through their
owner or fail closed before discarding the ownership manifest.
External nearest-neighbour queries use bounded geometric overfetch when stale, pending, or
wrong-model adapter hits are rejected during SQL hydration. This prevents obsolete top-k extension
results from crowding live matches out while retaining a fixed upper scan bound.
SQLite always resolves to the built-in
sqlite-vecadapter. Postgres defaults topgvectorandmay select an installed extension by name. Missing, duplicate, incompatible, or incorrectly scoped
extensions fail explicitly instead of silently falling back.
After updating to current
main, vector retrieval continues through the shared_run_vector_queryhook used by the reranker candidate-window logic from #1143. The hydrationsession defers its connection checkout until the adapter search finishes, so SQLite's single-
connection test/runtime pool does not deadlock while both extension-backed retrieval and stable
reranker pagination remain intact.
Latest-head review hardening separates the stable external storage key from mutable embedding
schema metadata, includes PostgreSQL user and explicit
search_pathidentity in the credential-free database namespace, and makes external cleanup failures preserve their authoritative
manifest by default. Cross-adapter migrations now document the required old-owner cleanup before
configuration changes instead of promising an unsafe automatic switch.
Full project indexing now forwards the configured external vector cleaner into its maintenance
store, matching watch-event indexing. Deleted or replaced entities therefore clean their external
vectors through the owning adapter instead of aborting when their manifest owner is not pgvector.
Embedding completions now verify and serialize the source generation through adapter persistence
and ready-state publication, so overlapping syncs cannot replace a newer stable-key vector with an
obsolete result. Built-in pgvector and sqlite-vec storage also retain the source hash for
defense-in-depth search and reconciliation checks. Pgvector storage probes are limited to schemas
on the active search path, and external namespaces include query-string Postgres hosts and ports
used for Unix-socket or multi-host connections.
This does not include a Milvus implementation. The intended follow-up is for the contributor and
Zilliz team to carry the backend in a separately versioned package built on this contract. Feedback
from @zc277584121 on whether this supports the package split proposed in #1041 would be especially
helpful.
Testing
Automated
git show --check --oneline --no-renames HEAD: passed.just fast-check: passed; type checking reported nine existing Python 3.14 asyncio deprecationwarnings.
uv run pytest -qacross the 18 repository semantic-vector, embedding-provider, reranker,pgvector, sqlite-vec, hybrid, pagination, threshold, and cleanup test modules: 319 passed.
uv run pytest -qacross the focused external cleanup, semantic search, semantic sync, harddelete, and project-service operation modules: 65 passed.
uv run pytest -q tests/services/test_project_service.py tests/services/test_project_service_operations.py tests/services/test_project_removal_bug.py:50 passed, 2 skipped.
uv run pytest -q tests/repository/test_semantic_search_base.py tests/repository/test_sqlite_vector_search_repository.py:57 passed, including PostgreSQL and SQLite external-reconciliation serialization.
BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -q tests/services/test_project_service_embedding_status.py:15 passed.
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -q tests/services/test_project_service_embedding_status.py:13 passed, 2 skipped.
BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -q test-int/test_embedding_status_vec0.py:passed, including a fresh connection that reloads sqlite-vec before verifying the physical row.
BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 .venv/bin/pytest -q --no-cov tests/index/test_local_project_vector_cleaner_wiring.py tests/repository/test_external_vector_cleanup.py: 3 passed.BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 .venv/bin/pytest -q --no-cov tests/index/test_local_project_index.py: 43 passed.BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 just doctor: passed, including real localembedding generation and sqlite-vec indexing.
582b94aa14847c8c3de91de0009691c1f1b742e8: passed acrossstatic checks, semantic tests, SQLite and PostgreSQL unit/integration matrices, CodeQL, DCO,
CLA, and PR-title validation.
BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -q --no-cov -m semantic test-int/semantic/test_search_diagnostics.py::test_similarity_formula_analysis: passed after updating the diagnostic to assert the adapter's normalized similarity contract instead of reading backend-native distances.BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 just fast-test: two selected local testspassed and one test skipped; six selected semantic integration cases errored during fixture
setup because Docker is not running locally, so no local Postgres result is claimed.
Manual
signed off.
mainat7402efc3into the branch and resolved the five additivevector-index/reranker conflicts without combining the feature boundaries.
Risks / Follow-ups
first external Milvus package validates it.
this PR makes vector storage replaceable but does not add such a provider.