Repository navigation
refactor(nodes): rename vector stores to store_* and graph-first db to graph_* - #1636
Conversation
🤖 Internal: Discord sync markerAuto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete. |
📝 WalkthroughWalkthroughAdds ArangoDB and HydraDB graph nodes, relocates vector/search nodes under ChangesGraph database nodes
Store package implementations
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
How about using directories for organization instead of prefixes? Instead of: Just like tha:t This would also free up the root |
There was a problem hiding this comment.
Actionable comments posted: 32
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@nodes/src/nodes/graph_hydradb/requirements.txt`:
- Line 1: Update the requests dependency entry in requirements.txt to require
the safe minimum version requests>=2.34.2, preserving the existing dependency
declaration format.
In `@nodes/src/nodes/vectordb_astra/astra_db.py`:
- Around line 380-385: Update the object deletion logic in flush() to issue one
collection.delete_many call using an $in filter over list(objectIds.keys()),
replacing the per-object loop while preserving the existing no-objectIds guard.
- Around line 386-434: Update the local is_valid_vector filter in flush so the
schema control document is accepted regardless of its near-zero vector
magnitude, while retaining the existing magnitude threshold for all other
documents. Use the document’s metadata/objectId to identify the schema control
document and preserve its insertion through valid_docs.
In `@nodes/src/nodes/vectordb_astra/README.md`:
- Around line 81-99: Regenerate the params section in the renamed vectordb_astra
module using nodes:docs-generate instead of editing README.md manually. Ensure
the generated field keys and View source link reference vectordb_astra
consistently, while preserving the generated block markers and content outside
the block.
- Around line 1-3: Update the README heading and node title from astra_db to
vectordb_astra, keeping the existing description and documentation content
unchanged.
In `@nodes/src/nodes/vectordb_atlas/atlas.py`:
- Around line 392-441: Update render() to avoid preallocating text with
renderChunkSize; build the rendered chunk content dynamically from the documents
actually returned while preserving chunk ordering and pagination behavior. Keep
the existing metadata validation, offset bounds, callback invocation, and
termination conditions intact, and ensure sparse chunk IDs still produce the
correct concatenated output.
- Around line 270-276: Initialize or otherwise handle results in the
vector-search aggregation flow around self.collection.aggregate(pipeline) so an
OperationFailure cannot reach _convertToDocs with an unassigned value. Preserve
the original Atlas error by re-raising it after logging, or return an
appropriate empty result if that is the established method contract.
In `@nodes/src/nodes/vectordb_atlas/IGlobal.py`:
- Around line 56-62: Update the subKey construction in IGlobal using
self.store.database and self.store.collection so it extracts their name strings
rather than interpolating pymongo objects; prefer the store’s database_name and
collection_name fields when available, matching the name-based approach in
vectordb_astra/IGlobal.py, while preserving the host/database/collection format.
- Around line 83-86: Update validateConfig in IGlobal.py to remove the
api_key/apikey validation guard, since Store.__init__ and services.json do not
define or consume that field. Ensure validation continues to the existing host,
database, and collection checks when the field is absent.
In `@nodes/src/nodes/vectordb_atlas/README.md`:
- Line 1: Update the hand-authored Markdown heading in the README from “atlas”
to the renamed node name “vectordb_atlas.” Do not modify anything inside the
ROCKETRIDE:GENERATED:PARAMS block, including its generated links or content.
In `@nodes/src/nodes/vectordb_chroma/chroma.py`:
- Around line 438-462: Fix payload-size batching in the chunk loop by replacing
shallow sys.getsizeof(tmp_struct) accounting with an estimate based on the
actual content, embedding, and metadata payload sizes. Since sum_size already
includes the current item, change the payload threshold check to compare
sum_size directly against self.payload_limit while preserving the 500-item flush
condition.
- Around line 364-393: Update getPaths to filter chunk-zero records using the
stored metadata key chunkId instead of chunk, while preserving the existing
parent filter, query, and path-building behavior.
- Around line 227-262: Correct the score calculation in _convertToDocs so lower
Chroma distances produce higher similarity scores for cosine, l2, and ip
metrics. Update both similarity branches and ensure the score < 0.20 filtering
uses the corrected score ordering.
- Around line 483-529: Update markDeleted and markActive to handle
collectionObj.get’s parallel-array response rather than iterating the result
dictionary. Pair each returned ID with its metadata, set isDeleted to True or
False respectively, and call collectionObj.update using the supported ids= and
metadatas= arguments; preserve the existing collection-existence and empty-input
early returns.
In `@nodes/src/nodes/vectordb_chroma/README.md`:
- Around line 95-117: Regenerate the README’s generated documentation using the
nodes:docs-generate task so the Source link for the vectordb_chroma node targets
nodes/src/nodes/vectordb_chroma. Do not edit the generated block manually.
In `@nodes/src/nodes/vectordb_elasticsearch/elasticsearch_store.py`:
- Around line 382-383: In the semantic-search validation around
docFilter.offset, replace BaseException with Exception so the non-zero offset
error is handled by standard callers' exception paths. Preserve the existing
error message and validation behavior.
- Around line 583-591: Update the batching logic around the action accumulation
and payload-limit check to estimate each document’s serialized request size
rather than using sys.getsizeof(doc). Serialize the action using the same or
equivalent encoding used for the Elasticsearch bulk request, add its byte length
to sum_size, and retain the existing 500-action and payload-limit flush
behavior.
In `@nodes/src/nodes/vectordb_elasticsearch/IInstance.py`:
- Around line 314-320: Move the client.ensure_index_vector call out of the
per-document loop in the vector ingestion flow, placing it once before iteration
after validating that vector_dim is positive. Keep the existing embedding and
vector_dim skip behavior inside the loop, and use the constant vector_dim for
the single index initialization.
In `@nodes/src/nodes/vectordb_elasticsearch/requirements.txt`:
- Around line 1-3: Update the node dependency list in requirements.txt by adding
requests with a safe minimum constraint of requests>=2.34.2, while preserving
the existing elasticsearch, opensearch-py, and numpy dependencies.
In `@nodes/src/nodes/vectordb_milvus/IGlobal.py`:
- Around line 65-90: Update the token-based cloud branch in the validation
connection flow to construct the URI with the same https:// scheme and host/port
shape used by Store.__init__. Keep host_raw and port unchanged for local
connections, and continue passing the resulting URI to connections.connect.
In `@nodes/src/nodes/vectordb_milvus/milvus.py`:
- Around line 470-478: Update the delete error handling in addChunks so failures
from self.client.delete are logged and then re-raised, matching the behavior in
remove(). Do not continue inserting replacement chunks when deletion of existing
objectId chunks fails; preserve the current successful deletion and insertion
flow.
- Around line 663-674: Update the result-processing loop in render() to read
chunkId from each point’s meta JSON, matching getPaths(), instead of accessing a
non-existent top-level field. Add the upper-bound guard so chunks at or beyond
offset + self.renderChunkSize are skipped before indexing text, while preserving
the existing lower-bound check and rendering behavior for valid chunks.
In `@nodes/src/nodes/vectordb_pinecone/IGlobal.py`:
- Around line 177-182: Update the collection lookup and mode validation in the
Pinecone initialization flow to use IndexModel attributes: compare `index.name`
when finding `existing_collection`, then inspect `existing_collection.spec` for
serverless mode instead of calling `.get()` on the model. Preserve the existing
compatibility-check behavior.
In `@nodes/src/nodes/vectordb_pinecone/pinecone.py`:
- Around line 238-245: Update searchKeyword so it does not pass the already
converted filter dict to get(), since get() unconditionally calls _convertFilter
and expects a DocFilter. Preserve the keyword predicate by either passing the
original DocFilter and applying the content condition within the query path, or
adding an explicit raw-filter path to get() that supports the converted filter.
- Around line 288-290: Fix the threshold attribute naming in the Pinecone node:
rename the class attribute threshhold_search to threshold_search and update the
points_satisfy_threshold filter to read self.threshold_search, matching the
value initialized by __init__ so semantic search applies the configured score
threshold.
In `@nodes/src/nodes/vectordb_pinecone/README.md`:
- Around line 54-59: Reconcile the Pinecone profile default documentation with
the schema by updating the conflicting default references in the README,
including the profile table and the statement below it, so they consistently
identify pod-based as the default.
In `@nodes/src/nodes/vectordb_qdrant/qdrant.py`:
- Around line 140-149: Update Qdrant’s __del__ method to close the existing
self.client before clearing it, while safely handling the client being unset or
unavailable. Preserve the current attribute-reset behavior after cleanup.
In `@nodes/src/nodes/vectordb_weaviate/IGlobal.py`:
- Around line 71-154: Update validateConfig to determine cloud versus local
using the configured mode value, matching the mode selection used by
Store.__init__ in weaviate.py, instead of inferring it from host patterns.
Preserve the existing cloud HTTP probe and local SDK/gRPC readiness checks, but
route them according to the selected mode so save-time validation matches
runtime behavior.
In `@nodes/src/nodes/vectordb_weaviate/weaviate.py`:
- Around line 489-541: Update markDeleted and markActive to use Weaviate’s batch
API, consistent with addChunks, instead of calling collectionObj.data.update
once per point. Batch the matching point UUIDs with the appropriate isDeleted
value while preserving the existing collection checks, empty-input handling, and
filtering behavior.
- Around line 543-606: Update the fetch_objects call inside render to pass
limit=self.renderChunkSize, ensuring each chunk window requests the full
configured batch rather than Weaviate’s default page size. Preserve the existing
offset filtering, ordering, callback behavior, and termination logic.
In `@nodes/test/vectordb_astra/test_convert_filter.py`:
- Line 15: Update the Store import in test_convert_filter.py to use the
relocated nodes.vectordb_astra module path, matching the package structure used
by the moved implementation. Remove the stale nodes.astra_db.astra_db reference
so the test can import Store and exercise _convertFilter.
In `@nodes/test/vectordb_milvus/test_addchunks_return.py`:
- Around line 19-23: Update the import inside
test_milvus_addchunks_returns_none_when_create_collection_fails to reference the
current Milvus module path after the package move, removing the stale
nodes.milvus.milvus target so the test can run when its skip is removed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 48b786df-7ae7-47b9-a373-642deb5e5c99
⛔ Files ignored due to path filters (11)
nodes/src/nodes/graph_arango/arango.svgis excluded by!**/*.svgnodes/src/nodes/graph_hydradb/hydradb.svgis excluded by!**/*.svgnodes/src/nodes/vectordb_astra/astra_db.svgis excluded by!**/*.svgnodes/src/nodes/vectordb_atlas/mongodb.svgis excluded by!**/*.svgnodes/src/nodes/vectordb_chroma/chroma.svgis excluded by!**/*.svgnodes/src/nodes/vectordb_elasticsearch/elasticsearch.svgis excluded by!**/*.svgnodes/src/nodes/vectordb_elasticsearch/opensearch.svgis excluded by!**/*.svgnodes/src/nodes/vectordb_milvus/milvus.svgis excluded by!**/*.svgnodes/src/nodes/vectordb_pinecone/pinecone.svgis excluded by!**/*.svgnodes/src/nodes/vectordb_qdrant/qdrant.svgis excluded by!**/*.svgnodes/src/nodes/vectordb_weaviate/weaviate.svgis excluded by!**/*.svg
📒 Files selected for processing (106)
nodes/src/nodes/graph_arango/IGlobal.pynodes/src/nodes/graph_arango/IInstance.pynodes/src/nodes/graph_arango/README.mdnodes/src/nodes/graph_arango/__init__.pynodes/src/nodes/graph_arango/requirements.txtnodes/src/nodes/graph_arango/services.jsonnodes/src/nodes/graph_arango/utils.pynodes/src/nodes/graph_hydradb/IGlobal.pynodes/src/nodes/graph_hydradb/IInstance.pynodes/src/nodes/graph_hydradb/README.mdnodes/src/nodes/graph_hydradb/__init__.pynodes/src/nodes/graph_hydradb/hydradb_client.pynodes/src/nodes/graph_hydradb/requirements.txtnodes/src/nodes/graph_hydradb/services.jsonnodes/src/nodes/vectordb_astra/IEndpoint.pynodes/src/nodes/vectordb_astra/IGlobal.pynodes/src/nodes/vectordb_astra/IInstance.pynodes/src/nodes/vectordb_astra/README.mdnodes/src/nodes/vectordb_astra/__init__.pynodes/src/nodes/vectordb_astra/astra_db.pynodes/src/nodes/vectordb_astra/requirements.txtnodes/src/nodes/vectordb_astra/services.jsonnodes/src/nodes/vectordb_atlas/IEndpoint.pynodes/src/nodes/vectordb_atlas/IGlobal.pynodes/src/nodes/vectordb_atlas/IInstance.pynodes/src/nodes/vectordb_atlas/README.mdnodes/src/nodes/vectordb_atlas/__init__.pynodes/src/nodes/vectordb_atlas/atlas.pynodes/src/nodes/vectordb_atlas/requirements.txtnodes/src/nodes/vectordb_atlas/services.jsonnodes/src/nodes/vectordb_chroma/IEndpoint.pynodes/src/nodes/vectordb_chroma/IGlobal.pynodes/src/nodes/vectordb_chroma/IInstance.pynodes/src/nodes/vectordb_chroma/README.mdnodes/src/nodes/vectordb_chroma/__init__.pynodes/src/nodes/vectordb_chroma/chroma.pynodes/src/nodes/vectordb_chroma/requirements.txtnodes/src/nodes/vectordb_chroma/services.jsonnodes/src/nodes/vectordb_elasticsearch/IEndpoint.pynodes/src/nodes/vectordb_elasticsearch/IGlobal.pynodes/src/nodes/vectordb_elasticsearch/IInstance.pynodes/src/nodes/vectordb_elasticsearch/README.mdnodes/src/nodes/vectordb_elasticsearch/__init__.pynodes/src/nodes/vectordb_elasticsearch/constants.pynodes/src/nodes/vectordb_elasticsearch/elasticsearch/README.mdnodes/src/nodes/vectordb_elasticsearch/elasticsearch_store.pynodes/src/nodes/vectordb_elasticsearch/opensearch/README.mdnodes/src/nodes/vectordb_elasticsearch/opensearch_client.pynodes/src/nodes/vectordb_elasticsearch/requirements.txtnodes/src/nodes/vectordb_elasticsearch/services.elasticsearch.jsonnodes/src/nodes/vectordb_elasticsearch/services.opensearch.jsonnodes/src/nodes/vectordb_milvus/IEndpoint.pynodes/src/nodes/vectordb_milvus/IGlobal.pynodes/src/nodes/vectordb_milvus/IInstance.pynodes/src/nodes/vectordb_milvus/README.mdnodes/src/nodes/vectordb_milvus/__init__.pynodes/src/nodes/vectordb_milvus/milvus.pynodes/src/nodes/vectordb_milvus/requirements.txtnodes/src/nodes/vectordb_milvus/services.jsonnodes/src/nodes/vectordb_pinecone/IEndpoint.pynodes/src/nodes/vectordb_pinecone/IGlobal.pynodes/src/nodes/vectordb_pinecone/IInstance.pynodes/src/nodes/vectordb_pinecone/README.mdnodes/src/nodes/vectordb_pinecone/__init__.pynodes/src/nodes/vectordb_pinecone/pinecone.pynodes/src/nodes/vectordb_pinecone/requirements.txtnodes/src/nodes/vectordb_pinecone/services.jsonnodes/src/nodes/vectordb_qdrant/IEndpoint.pynodes/src/nodes/vectordb_qdrant/IGlobal.pynodes/src/nodes/vectordb_qdrant/IInstance.pynodes/src/nodes/vectordb_qdrant/README.mdnodes/src/nodes/vectordb_qdrant/__init__.pynodes/src/nodes/vectordb_qdrant/qdrant.pynodes/src/nodes/vectordb_qdrant/requirements.txtnodes/src/nodes/vectordb_qdrant/services.jsonnodes/src/nodes/vectordb_weaviate/IEndpoint.pynodes/src/nodes/vectordb_weaviate/IGlobal.pynodes/src/nodes/vectordb_weaviate/IInstance.pynodes/src/nodes/vectordb_weaviate/README.mdnodes/src/nodes/vectordb_weaviate/__init__.pynodes/src/nodes/vectordb_weaviate/requirements.txtnodes/src/nodes/vectordb_weaviate/services.jsonnodes/src/nodes/vectordb_weaviate/weaviate.pynodes/test/conftest.pynodes/test/ocr/test_model_server_ocr.pynodes/test/test_graph_arango.pynodes/test/test_graph_hydradb.pynodes/test/vectordb_astra/__init__.pynodes/test/vectordb_astra/test_convert_filter.pynodes/test/vectordb_atlas/__init__.pynodes/test/vectordb_atlas/test_isdeleted_filter.pynodes/test/vectordb_chroma/__init__.pynodes/test/vectordb_chroma/test_coerce_port.pynodes/test/vectordb_chroma/test_convert_filter_explicit_none.pynodes/test/vectordb_milvus/__init__.pynodes/test/vectordb_milvus/test_addchunks_return.pynodes/test/vectordb_milvus/test_filter_injection.pynodes/test/vectordb_milvus/test_search_semantic.pynodes/test/vectordb_pinecone/__init__.pynodes/test/vectordb_pinecone/test_record_updates.pynodes/test/vectordb_qdrant/__init__.pynodes/test/vectordb_qdrant/test_mark_deleted.pynodes/test/vectordb_qdrant/test_search_threshold.pynodes/test/vectordb_weaviate/__init__.pynodes/test/vectordb_weaviate/test_record_updates.pypackages/docs/redirects.ts
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
🛑 Comments failed to post (32)
nodes/src/nodes/graph_hydradb/requirements.txt (1)
1-1: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Pin
requeststo a safe minimum version.To avoid resolving vulnerable older versions of
requestsand to maintain accurate dependency/vulnerability scans, please pinrequeststo a safe minimum version. Based on learnings, it is recommended to userequests>=2.34.2in the noderequirements.txtfiles.🔒️ Proposed fix to pin requests
-requests +requests>=2.34.2📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.requests>=2.34.2🧰 Tools
🪛 OSV Scanner (2.4.0)
[HIGH] 1-1: requests 2.9.2: undefined
(PYSEC-2018-28)
[HIGH] 1-1: requests 2.9.2: undefined
(PYSEC-2023-74)
[HIGH] 1-1: requests 2.9.2: Requests vulnerable to .netrc credentials leak via malicious URLs
(PYSEC-2026-1872)
[HIGH] 1-1: requests 2.9.2: Requests
Sessionobject does not verify requests after making first request with verify=False(PYSEC-2026-1873)
[HIGH] 1-1: requests 2.9.2: undefined
(PYSEC-2026-2275)
[HIGH] 1-1: requests 2.9.2: Requests vulnerable to .netrc credentials leak via malicious URLs
(GHSA-9hjg-9r4m-mvj7)
[HIGH] 1-1: requests 2.9.2: Requests
Sessionobject does not verify requests after making first request with verify=False(GHSA-9wx4-h78v-vm56)
[HIGH] 1-1: requests 2.9.2: Requests has Insecure Temp File Reuse in its extract_zipped_paths() utility function
(GHSA-gc5v-m9x4-r6x2)
[HIGH] 1-1: requests 2.9.2: Unintended leak of Proxy-Authorization header in requests
(GHSA-j8r2-6x86-q33q)
[HIGH] 1-1: requests 2.9.2: Insufficiently Protected Credentials in Requests
(GHSA-x84v-xcm2-53pg)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/graph_hydradb/requirements.txt` at line 1, Update the requests dependency entry in requirements.txt to require the safe minimum version requests>=2.34.2, preserving the existing dependency declaration format.Source: Learnings
nodes/src/nodes/vectordb_astra/astra_db.py (2)
380-385: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
Consider batching the per-object deletes into a single call.
flush()issues onedelete_many({'meta.objectId': objectId})call per objectId in the batch instead of a singledelete_many({'meta.objectId': {'$in': list(objectIds.keys())}}). For batches touching many distinct objects this is N round-trips instead of 1.♻️ Proposed fix
- if len(objectIds): - # Delete all documents with these object IDs - for objectId in objectIds.keys(): - collection.delete_many({'meta.objectId': objectId}) + if len(objectIds): + # Delete all documents with these object IDs in one round-trip + collection.delete_many({'meta.objectId': {'$in': list(objectIds.keys())}})📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.# First, delete existing documents if we have objectIds to delete if len(objectIds): # Delete all documents with these object IDs in one round-trip collection.delete_many({'meta.objectId': {'$in': list(objectIds.keys())}})🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_astra/astra_db.py` around lines 380 - 385, Update the object deletion logic in flush() to issue one collection.delete_many call using an $in filter over list(objectIds.keys()), replacing the per-object loop while preserving the existing no-objectIds guard.
386-434: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
Schema control-document vector is likely always filtered out by
is_valid_vector, breaking control-doc persistence.The zero-vector workaround (line 417-418) sets the
schemacontrol doc's embedding to[1e-8] * len(embedding), butflush()'sis_valid_vector(line 386-388) requiresmagnitude > 1e-6. For any realistic embedding dimension (e.g. 384–3072),magnitude = 1e-8 * sqrt(dim)stays well below1e-6(e.g. ≈3.9e-7 for dim=1536), so the control document is silently excluded fromvalid_docsand never inserted. SincedoesCollectionExist()'s docstring explicitly relies on a "control document" for validation, this defeats the very workaround meant to let the schema doc pass the ingest path, causing the base class's collection validation to always treat a freshly created collection as missing its control document.🐛 Proposed fix: exempt the schema control document from the magnitude filter
- def is_valid_vector(vector): - magnitude = sum(x * x for x in vector) ** 0.5 - return magnitude > 1e-6 - - valid_docs = [doc for doc in documents if is_valid_vector(doc.get('$vector', [1]))] + def is_valid_vector(doc): + # The schema control document intentionally uses a near-zero + # vector; never drop it based on magnitude. + if doc.get('meta', {}).get('objectId') == 'schema': + return True + vector = doc.get('$vector', [1]) + magnitude = sum(x * x for x in vector) ** 0.5 + return magnitude > 1e-6 + + valid_docs = [doc for doc in documents if is_valid_vector(doc)]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def is_valid_vector(doc): # The schema control document intentionally uses a near-zero # vector; never drop it based on magnitude. if doc.get('meta', {}).get('objectId') == 'schema': return True vector = doc.get('$vector', [1]) magnitude = sum(x * x for x in vector) ** 0.5 return magnitude > 1e-6 valid_docs = [doc for doc in documents if is_valid_vector(doc)] # Insert the new documents if len(valid_docs): collection.insert_many(valid_docs) # Clear them objectIds = {} documents = [] # For each document, check if we need to delete existing chunks for chunk in chunks: # If we are writing chunk 0, then delete the object if not chunk.metadata.chunkId: # Save this object id to be deleted objectIds[chunk.metadata.objectId] = True # For each document, prepare the data for insertion for chunk in chunks: # Get the embedding embedding = chunk.embedding # If we do not have an embedding if embedding is None: raise Exception('No embedding in document') # Handle control document with zero vectors if chunk.metadata.objectId == 'schema' and all(x == 0 for x in embedding): embedding = [1e-8] * len(embedding) # Create the document structure for AstraDB document = { '_id': str(uuid4()), '$vector': DataAPIVector(embedding), 'content': chunk.page_content, 'meta': chunk.metadata.__dict__, # Convert metadata to dict } documents.append(document) if len(documents) >= 500: flush() # Flush any remaining documents flush()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_astra/astra_db.py` around lines 386 - 434, Update the local is_valid_vector filter in flush so the schema control document is accepted regardless of its near-zero vector magnitude, while retaining the existing magnitude threshold for all other documents. Use the document’s metadata/objectId to identify the schema control document and preserve its insertion through valid_docs.nodes/src/nodes/vectordb_astra/README.md (2)
1-3: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Title still references the old node name.
The heading (
# astra_db) and node title weren't updated tovectordb_astraas part of this rename, unlike the folder/module path itself.✏️ Proposed fix
-# astra_db +# vectordb_astra📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.# vectordb_astra A RocketRide store node that persists embedded documents in DataStax Astra DB and retrieves them by semantic or keyword search.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_astra/README.md` around lines 1 - 3, Update the README heading and node title from astra_db to vectordb_astra, keeping the existing description and documentation content unchanged.
81-99: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Generated params section still references the pre-rename node name/path.
Field keys (
astra_db.api_endpoint,astra_db.application_token,astra_db.provider) and the "View source" link (.../nodes/src/nodes/astra_db) are stale post-rename. Per path instructions, this block must not be hand-edited (it's regenerated bynodes:docs-generate), so the fix is to re-run the doc generator against the renamedvectordb_astramodule rather than editing these lines directly.As per path instructions,
nodes/src/nodes/*/README.md: "do not hand-edit content between<!-- ROCKETRIDE:GENERATED:PARAMS START -->and<!-- ROCKETRIDE:GENERATED:PARAMS END -->as it is regenerated bynodes:docs-generate."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_astra/README.md` around lines 81 - 99, Regenerate the params section in the renamed vectordb_astra module using nodes:docs-generate instead of editing README.md manually. Ensure the generated field keys and View source link reference vectordb_astra consistently, while preserving the generated block markers and content outside the block.Source: Path instructions
nodes/src/nodes/vectordb_atlas/atlas.py (2)
270-276: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
resultsmay be unbound if the vector-search aggregation fails.If
self.collection.aggregate(pipeline)raisesOperationFailure, theexceptblock only logs the error and falls through —resultsis never assigned, soself._convertToDocs(results, include_score=True)on line 276 raisesUnboundLocalErrorinstead of surfacing the original Atlas error.🐛 Proposed fix
try: results = list(self.collection.aggregate(pipeline)) logger.info(f'Results: {results}') except OperationFailure as e: logger.error(f'OperationFailure: {e}') + return [] return self._convertToDocs(results, include_score=True)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.try: results = list(self.collection.aggregate(pipeline)) logger.info(f'Results: {results}') except OperationFailure as e: logger.error(f'OperationFailure: {e}') return [] return self._convertToDocs(results, include_score=True)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_atlas/atlas.py` around lines 270 - 276, Initialize or otherwise handle results in the vector-search aggregation flow around self.collection.aggregate(pipeline) so an OperationFailure cannot reach _convertToDocs with an unassigned value. Preserve the original Atlas error by re-raising it after logging, or return an appropriate empty result if that is the established method contract.
392-441: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift
render()preallocates a list sized torenderChunkSize(default ~33.5M) on every call/iteration.
text = [''] * self.renderChunkSizewith the defaultrenderChunkSize = 32 * 1024 * 1024allocates a ~33.5-million-element list even for objects with a single chunk, since this happens unconditionally before any documents are inspected. This is a significant memory/latency cost on what looks like a routine rendering path. The same pattern appears in sibling providers (astra_db.py doesn't have it, but chroma.py/milvus.py/pinecone.py/weaviate.py do), suggestingrenderChunkSize's intended semantics (payload byte-size limit vs. chunk-count page size) may be conflated across the codebase.🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 403-403: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: self.collection.find(filter_conditions)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').(xpath-injection-python)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_atlas/atlas.py` around lines 392 - 441, Update render() to avoid preallocating text with renderChunkSize; build the rendered chunk content dynamically from the documents actually returned while preserving chunk ordering and pagination behavior. Keep the existing metadata validation, offset bounds, callback invocation, and termination conditions intact, and ensure sparse chunk IDs still produce the correct concatenated output.nodes/src/nodes/vectordb_atlas/IGlobal.py (2)
56-62: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win
subKeyis built from pymongoDatabase/Collectionobjects, not their names.
self.store.databaseandself.store.collectionare pymongoDatabase/Collectioninstances (peratlas.py'sself.database = self.client[self.database_name]/self.database[self.collection_name]), not name strings. Interpolating them intosubKeyyields their verboserepr()(e.g.Database(MongoClient(host=[...], ...), 'testdb')) instead of the intendedhost/database/collectionidentifier. Contrast withvectordb_astra/IGlobal.py, which explicitly guards against this by preferringcollection_nameovercollection.🐛 Proposed fix
# Get the info about our store - database = self.store.database - collection = self.store.collection + database = self.store.database_name + collection = self.store.collection_name host = self.store.host📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.# Get the info about our store database = self.store.database_name collection = self.store.collection_name host = self.store.host # Format it into a subKey subKey = f'{host}/{database}/{collection}'🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_atlas/IGlobal.py` around lines 56 - 62, Update the subKey construction in IGlobal using self.store.database and self.store.collection so it extracts their name strings rather than interpolating pymongo objects; prefer the store’s database_name and collection_name fields when available, matching the name-based approach in vectordb_astra/IGlobal.py, while preserving the host/database/collection format.
83-86: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail printf '\n== file list ==\n' git ls-files | rg '(^|/)(IGlobal\.py|atlas\.py|services\.json|README\.md|.*schema.*|.*services.*)$' || true printf '\n== locate relevant symbols ==\n' rg -n --hidden --glob '!**/.git/**' 'validateConfig|apikey|atlas.provider|vector.cloud.host|vector.database|Store\.__init__|class Store' nodes src . || true printf '\n== inspect IGlobal.py ==\n' sed -n '1,180p' nodes/src/nodes/vectordb_atlas/IGlobal.py printf '\n== inspect atlas.py ==\n' sed -n '1,220p' nodes/src/nodes/vectordb_atlas/atlas.py printf '\n== inspect services.json candidates ==\n' fd -a 'services\.json$|schema' . || trueRepository: rocketride-org/rocketride-server
Length of output: 50389
🏁 Script executed:
#!/bin/bash set -euo pipefail sed -n '1,180p' nodes/src/nodes/vectordb_atlas/IGlobal.py printf '\n----\n' sed -n '1,220p' nodes/src/nodes/vectordb_atlas/atlas.py printf '\n----\n' rg -n 'apikey|atlas.provider|vector.cloud.host|vector.database|validateConfig|services\.json' nodes/src/nodes/vectordb_atlas . --glob '!**/.git/**'Repository: rocketride-org/rocketride-server
Length of output: 50389
Remove the
apikeyguard or add the field to the schema —nodes/src/nodes/vectordb_atlas/services.jsonexposes onlyatlas.provider,vector.cloud.host, andvector.database, andStore.__init__never readsapikey. As written,validateConfigexits before host/database/collection checks for any config without that field.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_atlas/IGlobal.py` around lines 83 - 86, Update validateConfig in IGlobal.py to remove the api_key/apikey validation guard, since Store.__init__ and services.json do not define or consume that field. Ensure validation continues to the existing host, database, and collection checks when the field is absent.nodes/src/nodes/vectordb_atlas/README.md (1)
1-1: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Title still says
# atlas; should reflect the renamed node.This heading is outside the
<!-- ROCKETRIDE:GENERATED:PARAMS -->block, so it's hand-authored and wasn't updated along with the folder/module rename tovectordb_atlas.📝 Proposed fix
-# atlas +# vectordb_atlasAs per path instructions, the generated params block (including the stale "View source" link at line 76) should not be hand-edited, since it is regenerated by
nodes:docs-generate. Based on path instructions: "Documentation must be co-located with the code it documents ... do not hand-edit content between<!-- ROCKETRIDE:GENERATED:PARAMS START -->and<!-- ROCKETRIDE:GENERATED:PARAMS END -->as it is regenerated bynodes:docs-generate."📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.# vectordb_atlas🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_atlas/README.md` at line 1, Update the hand-authored Markdown heading in the README from “atlas” to the renamed node name “vectordb_atlas.” Do not modify anything inside the ROCKETRIDE:GENERATED:PARAMS block, including its generated links or content.Source: Path instructions
nodes/src/nodes/vectordb_chroma/chroma.py (4)
227-262: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
chromadb query distances cosine l2 ip meaning range💡 Result:
In ChromaDB, the distance metric defines how the similarity between vectors is calculated, with lower values indicating higher similarity [1][2]. You can configure this at the collection creation level using the
hnsw.spaceparameter [1][3]. The available options are l2, cosine, and ip [1][4][3]. Squared L2 (l2) - Definition: Squared Euclidean distance [1][4]. - Equation: d = Σ(Ai - Bi)² [4]. - Range: Starts at 0 (identical) and increases toward positive infinity [2]. - Usage: Measures absolute geometric distance, making it suitable when true spatial proximity is required [4]. It is the default metric in ChromaDB [1][4]. Cosine Similarity (cosine) - Definition: Measures the cosine distance between two vectors (often used for text embeddings) [4]. - Equation: d = 1.0 - (Σ(Ai * Bi) / (√Σ(Ai²) * √Σ(Bi²))) [4]. - Range: Ranges from 0 to 2. A distance of 0 means the vectors are identical (cosine similarity = 1), and a distance of 2 means the vectors are opposite (cosine similarity = -1). - Usage: Focuses on the angle/direction between vectors while ignoring their magnitude [4]. Inner Product (ip) - Definition: Measures the inner product distance [4]. - Equation: d = 1.0 - Σ(Ai * Bi) [4]. - Range: Does not have a fixed upper bound; it depends on the magnitude of the vectors. - Usage: Often used in recommendation systems where both vector alignment and magnitude are important [4]. Important Note: L2 and IP are sensitive to the magnitude of the vectors [2]. It is recommended to normalize your embeddings before adding them to ChromaDB if you are using these metrics [2]. Note that the distance metric cannot be changed after a collection has been created [1][4][3].Citations:
- 1: https://cookbook.chromadb.dev/core/collections/
- 2: https://cookbook.chromadb.dev/faq/
- 3: https://cookbook.chromadb.dev/core/configuration/
- 4: https://docs.trychroma.com/docs/collections/configure
Score mapping is reversed for Chroma distances.
Chroma reports lower distances for better matches acrosscosine,l2, andip, but both branches here turn larger distances into larger scores. That ranks worse hits above better ones and makes thescore < 0.20cutoff ineffective.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_chroma/chroma.py` around lines 227 - 262, Correct the score calculation in _convertToDocs so lower Chroma distances produce higher similarity scores for cosine, l2, and ip metrics. Update both similarity branches and ensure the score < 0.20 filtering uses the corrected score ordering.
364-393: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
getPathsfilters on'chunk'instead of'chunkId'.Every other filter in this file (
_convertFilter,render) consistently keys chunk-index metadata as'chunkId'.getPaths's base filter usesfilter_dict = {'chunk': 0}— a field that doesn't exist in stored metadata, so the ChromaDBwherefilter will not match chunk-0 records, andgetPathswill effectively always return{}.🐛 Proposed fix
- filter_dict = {'chunk': 0} + filter_dict = {'chunkId': {'$eq': 0}}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def getPaths(self, parent: str | None = None, offset: int = 0, limit: int = 1000) -> Dict[str, str]: """ Retrieve unique parent paths. """ # If the collection does not exists, by definition there are # no paths to return if not self.doesCollectionExist(): return {} # Base filter: Only chunk 0 filter_dict = {'chunkId': {'$eq': 0}} # If parent specified, add condition if parent is not None: filter_dict['parent'] = {'$eq': json.dumps(parent, ensure_ascii=False)} # Perform the query results = self.collectionObj.get(where=filter_dict, limit=limit, offset=offset) # Build paths dictionary paths = {} for i in range(len(results['ids'])): metadata = results['metadatas'][i] parent_id = metadata.get('parent', None) object_id = metadata.get('objectId', None) if parent_id and object_id: paths[parent_id] = object_id return paths🧰 Tools
🪛 ast-grep (0.44.1)
[info] 377-377: use jsonify instead of json.dumps for JSON output
Context: json.dumps(parent, ensure_ascii=False)
Note: [CWE-116] Improper Encoding or Escaping of Output.(use-jsonify)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_chroma/chroma.py` around lines 364 - 393, Update getPaths to filter chunk-zero records using the stored metadata key chunkId instead of chunk, while preserving the existing parent filter, query, and path-building behavior.
438-462: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
Payload-size batching is effectively broken.
sys.getsizeof(tmp_struct)(line 456) only measures the shallow size of the dict container itself, not the nestedcontentstring,metadict, orvectorlist it references — socur_sizestays roughly constant (a few hundred bytes) regardless of actual embedding/document size, making thesum_size + cur_size > self.payload_limitcheck almost never trigger. Large batches of sizeable embeddings/content can therefore accumulate well pastpayload_limitbefore a flush happens, relying solely on the 500-item count cap. Separately,sum_sizeis already incremented bycur_sizeon line 457 before the check on line 459 addscur_sizeagain, double-counting the current item in the threshold comparison.Consider computing size from the actual serialized payload (e.g.,
len(content.encode()),len(embedding) * 4for float32, plus a JSON-encoded metadata estimate) rather thansys.getsizeofon the wrapper dict, and drop the redundant+ cur_sizein the comparison sincesum_sizealready includes it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_chroma/chroma.py` around lines 438 - 462, Fix payload-size batching in the chunk loop by replacing shallow sys.getsizeof(tmp_struct) accounting with an estimate based on the actual content, embedding, and metadata payload sizes. Since sum_size already includes the current item, change the payload threshold check to compare sum_size directly against self.payload_limit while preserving the 500-item flush condition.
483-529: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
chromadb python client Collection.update method signature ids metadatas where💡 Result:
The ChromaDB Python client
Collection.updatemethod does not accept awhereclause parameter [1]. Thewhereparameter is used for filtering during queries or deletions (e.g., in.get(),.query(), or.delete()), but not for updating specific records [2][3]. TheCollection.updatemethod is designed to update existing records by their specific IDs [2][1]. The method signature is as follows [1]: collection.update( ids: OneOrMany[ID], embeddings: Optional[OneOrMany[Embedding]] = None, metadatas: Optional[OneOrMany[Metadata]] = None, documents: Optional[OneOrMany[Document]] = None, images: Optional[OneOrMany[Image]] = None, uris: Optional[OneOrMany[URI]] = None) Key points: -ids: Required. These are the unique identifiers of the records you wish to update [2][1]. If an ID provided does not exist in the collection, an error will be logged and the update for that specific record will be ignored [4]. -metadatas: Optional. You can provide new metadata dictionaries to replace or update the existing metadata for the specified IDs [2][1]. - Nowhereclause: Because this method targets records by their explicit ID, it does not support filtering records via awhereclause [2][1]. To update records based on a filter, you must first retrieve the IDs using a method like.get(where=...)and then pass those IDs to.update()[2][1]. When providing multiple values (like a list of IDs and a corresponding list of metadata), they must be of the same length, as the data is processed in a columnar format where the index of the element corresponds to the record [2][1].Citations:
- 1: https://github.com/chroma-core/chroma/blob/main/chromadb/api/models/Collection.py
- 2: https://docs.trychroma.com/reference/python/collection
- 3: https://chroma-core-chroma.mintlify.app/api/python/collection
- 4: https://docs.trychroma.com/docs/collections/update-data
🏁 Script executed:
#!/bin/bash set -euo pipefail # Inspect the target file around the reported lines. file='nodes/src/nodes/vectordb_chroma/chroma.py' wc -l "$file" sed -n '430,560p' "$file" # Find any other Chroma collection update/get usages for context. rg -n "collectionObj\.(get|update)\(" nodes/src/nodes/vectordb_chroma -n # Inspect package/dependency pins that may indicate chromadb version. rg -n "chromadb|chroma" -g 'pyproject.toml' -g 'requirements*.txt' -g 'poetry.lock' -g 'uv.lock' -g 'Pipfile.lock' -g 'setup.cfg' -g 'setup.py' -g 'package.json' -g 'environment.yml' .Repository: rocketride-org/rocketride-server
Length of output: 5932
🏁 Script executed:
#!/bin/bash set -euo pipefail file='nodes/src/nodes/vectordb_chroma/chroma.py' # Show the class definition and how collectionObj is initialized. rg -n "collectionObj|chromadb|Collection" "$file" sed -n '1,220p' "$file" # Look for any local helper that might wrap Chroma's Collection methods. rg -n "def .*update|def .*get|class .*Collection|collectionObj =" nodes/src/nodes/vectordb_chromaRepository: rocketride-org/rocketride-server
Length of output: 12597
Fix
markDeleted/markActiveto update records by ID.
self.collectionObj.get(...)returns a dict of parallel arrays, so iteratingresultswalks keys, not records;update()also expectsids=andmetadatas=, notwhere=/new_metadata=. Both methods fail on any non-emptyobjectIds.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_chroma/chroma.py` around lines 483 - 529, Update markDeleted and markActive to handle collectionObj.get’s parallel-array response rather than iterating the result dictionary. Pair each returned ID with its metadata, set isDeleted to True or False respectively, and call collectionObj.update using the supported ids= and metadatas= arguments; preserve the existing collection-existence and empty-input early returns.nodes/src/nodes/vectordb_chroma/README.md (1)
95-117: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo '--- README excerpt ---' sed -n '90,125p' nodes/src/nodes/vectordb_chroma/README.md echo echo '--- node directories ---' find nodes/src/nodes -maxdepth 2 -type d | sort echo echo '--- possible chroma/vectordb_chroma files ---' find nodes/src/nodes -maxdepth 2 \( -path '*/chroma*' -o -path '*/vectordb_chroma*' \) | sortRepository: rocketride-org/rocketride-server
Length of output: 7907
🏁 Script executed:
#!/bin/bash set -euo pipefail sed -n '1,220p' nodes/src/nodes/vectordb_chroma/README.md | nl -ba | sed -n '90,125p' echo echo '--- directories/files ---' find nodes/src/nodes -maxdepth 2 \( -path '*/chroma*' -o -path '*/vectordb_chroma*' \) -print | sort echo echo '--- docs-generation hints ---' rg -n "ROCKETRIDE:GENERATED:PARAMS|View source|vectordb_chroma|nodes/chroma" nodes/src/nodes -SRepository: rocketride-org/rocketride-server
Length of output: 210
Regenerate the generated Source link. It still points to
nodes/src/nodes/chroma; rerunnodes:docs-generateso the README links tonodes/src/nodes/vectordb_chromainstead of editing the generated block by hand.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_chroma/README.md` around lines 95 - 117, Regenerate the README’s generated documentation using the nodes:docs-generate task so the Source link for the vectordb_chroma node targets nodes/src/nodes/vectordb_chroma. Do not edit the generated block manually.Source: Path instructions
nodes/src/nodes/vectordb_elasticsearch/elasticsearch_store.py (2)
382-383: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Raise
Exception, notBaseException.
BaseExceptionbypasses callers'except Exceptionhandlers and can crash the pipeline instead of being handled/formatted like other store errors.🐛 Proposed fix
- if docFilter.offset: - raise BaseException('Non-zero offset is not supported in semantic searching') + if docFilter.offset: + raise Exception('Non-zero offset is not supported in semantic searching')📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.if docFilter.offset: raise Exception('Non-zero offset is not supported in semantic searching')🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_elasticsearch/elasticsearch_store.py` around lines 382 - 383, In the semantic-search validation around docFilter.offset, replace BaseException with Exception so the non-zero offset error is handled by standard callers' exception paths. Preserve the existing error message and validation behavior.
583-591: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
sys.getsizeof(doc)does not measure serialized payload size, so thepayload_limitguard is effectively dead.
sys.getsizeofreturns only the shallow size of the outerdict(a few hundred bytes); it does not account for the nestedembeddinglist,page_content, ormeta. As a resultsum_sizestays tiny and thesum_size > self.payload_limitbranch never triggers — only the 500-action count flush applies. For large chunks this can let a single bulk request exceed Elasticsearch'shttp.max_content_length.Approximate the real payload size instead (e.g. via serialization length):
♻️ Suggested size estimate
+import json ... - cur_size = sys.getsizeof(doc) + cur_size = len(json.dumps(doc['_source'], default=str).encode('utf-8'))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.cur_size = len(json.dumps(doc['_source'], default=str).encode('utf-8')) sum_size += cur_size actions.append(doc) # Flush if we hit batch size or payload limit if (len(actions) >= 500) or (sum_size > self.payload_limit): _run_bulk(actions) actions = [] sum_size = 0🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_elasticsearch/elasticsearch_store.py` around lines 583 - 591, Update the batching logic around the action accumulation and payload-limit check to estimate each document’s serialized request size rather than using sys.getsizeof(doc). Serialize the action using the same or equivalent encoding used for the Elasticsearch bulk request, add its byte length to sum_size, and retain the existing 500-action and payload-limit flush behavior.nodes/src/nodes/vectordb_elasticsearch/IInstance.py (1)
314-320: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
Hoist
ensure_index_vectorout of the per-document loop.
ensure_index_vectorrunsindices.exists+get_mappingon every call, so invoking it once per document adds a network round-trip per chunk during ingestion.vector_dimis constant across the loop, so ensure the index once before iterating.♻️ Suggested restructure
vector_dim = self._os_get_vector_dim() debug(f'writeDocuments ingest count={len(documents)} index={index} mode=vstore') + if vector_dim <= 0: + debug('Vector mode missing vector_dim; cannot index') + return + client.ensure_index_vector(index=index, dimension=vector_dim) for doc in documents: ... - if vector_dim <= 0: - debug('Vector mode missing vector_dim; cannot index') - continue - client.ensure_index_vector(index=index, dimension=vector_dim)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.vector_dim = self._os_get_vector_dim() debug(f'writeDocuments ingest count={len(documents)} index={index} mode=vstore') if vector_dim <= 0: debug('Vector mode missing vector_dim; cannot index') return client.ensure_index_vector(index=index, dimension=vector_dim)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_elasticsearch/IInstance.py` around lines 314 - 320, Move the client.ensure_index_vector call out of the per-document loop in the vector ingestion flow, placing it once before iteration after validating that vector_dim is positive. Keep the existing embedding and vector_dim skip behavior inside the loop, and use the constant vector_dim for the single index initialization.nodes/src/nodes/vectordb_elasticsearch/requirements.txt (1)
1-3: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win
Pin
requeststo a safe minimum version.Based on learnings, please pin
requeststo a safe minimum version (e.g.,requests>=2.34.2) in the node'srequirements.txtto avoid resolving vulnerable older versions and to keep dependency scans accurate.📦 Proposed fix
elasticsearch>=8.0.0,<9.0.0 opensearch-py==3.2.0 numpy +requests>=2.34.2📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.elasticsearch>=8.0.0,<9.0.0 opensearch-py==3.2.0 numpy requests>=2.34.2🧰 Tools
🪛 OSV Scanner (2.4.0)
[MEDIUM] 1-1: idna 3.9.0: undefined
(PYSEC-2026-215)
[MEDIUM] 1-1: idna 3.9.0: Internationalized Domain Names in Applications (IDNA): Specially crafted inputs to idna.encode() can bypass CVE-2024-3651 fix
(GHSA-65pc-fj4g-8rjx)
[HIGH] 1-1: urllib3 1.26.20: undefined
(PYSEC-2026-141)
[HIGH] 1-1: urllib3 1.26.20: urllib3 streaming API improperly handles highly compressed data
(PYSEC-2026-1994)
[HIGH] 1-1: urllib3 1.26.20: Decompression-bomb safeguards bypassed when following HTTP redirects (streaming API)
(PYSEC-2026-1996)
[HIGH] 1-1: urllib3 1.26.20: urllib3 allows an unbounded number of links in the decompression chain
(PYSEC-2026-1998)
[HIGH] 1-1: urllib3 1.26.20: urllib3 redirects are not disabled when retries are disabled on PoolManager instantiation
(PYSEC-2026-1999)
[HIGH] 1-1: urllib3 1.26.20: urllib3 streaming API improperly handles highly compressed data
(GHSA-2xpw-w6gg-jr37)
[HIGH] 1-1: urllib3 1.26.20: Decompression-bomb safeguards bypassed when following HTTP redirects (streaming API)
(GHSA-38jv-5279-wg99)
[HIGH] 1-1: urllib3 1.26.20: urllib3 allows an unbounded number of links in the decompression chain
(GHSA-gm62-xv2j-4w53)
[HIGH] 1-1: urllib3 1.26.20: urllib3 redirects are not disabled when retries are disabled on PoolManager instantiation
(GHSA-pq67-6m6q-mj2v)
[HIGH] 1-1: urllib3 1.26.20: urllib3: Sensitive headers forwarded across origins in proxied low-level redirects
(GHSA-qccp-gfcp-xxvc)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_elasticsearch/requirements.txt` around lines 1 - 3, Update the node dependency list in requirements.txt by adding requests with a safe minimum constraint of requests>=2.34.2, while preserving the existing elasticsearch, opensearch-py, and numpy dependencies.Source: Learnings
nodes/src/nodes/vectordb_milvus/IGlobal.py (1)
65-90: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail printf '\n## IGlobal.py slice\n' sed -n '50,110p' nodes/src/nodes/vectordb_milvus/IGlobal.py printf '\n## milvus.py connection usages\n' rg -n "connections\.connect|uri=|https://|http://" nodes/src/nodes/vectordb_milvus -n printf '\n## milvus.py relevant slice\n' sed -n '1,220p' nodes/src/nodes/vectordb_milvus/milvus.pyRepository: rocketride-org/rocketride-server
Length of output: 13616
🏁 Script executed:
#!/bin/bash set -euo pipefail nl -ba nodes/src/nodes/vectordb_milvus/IGlobal.py | sed -n '1,130p'Repository: rocketride-org/rocketride-server
Length of output: 210
Cloud validation probe needs the same
https://URI as runtime —uri = f'{host_raw}:{port}'omits the scheme thatStore.__init__uses for cloud profiles, so token-based configs can fail validation with a false warning. Use the same cloud URI shape as the runtime path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_milvus/IGlobal.py` around lines 65 - 90, Update the token-based cloud branch in the validation connection flow to construct the URI with the same https:// scheme and host/port shape used by Store.__init__. Keep host_raw and port unchanged for local connections, and continue passing the resulting URI to connections.connect.nodes/src/nodes/vectordb_milvus/milvus.py (2)
470-478: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Silently swallowed delete failure risks duplicate/stale chunks on document replace.
Unlike
remove()(lines 538-543), which re-raises after logging aclient.deletefailure,addChunks()only logs viaengLib.debugand continues to insert the new chunks regardless of whether the old ones for the sameobjectIdwere actually removed. If the delete fails (timeout, transient error, etc.), old and new chunks for the sameobjectIdwill coexist, silently corrupting search results andrender()output (duplicated/interleaved content) with no signal to the caller.🐛 Proposed fix
try: # Delete entities self.client.delete(collection_name=self.collection, filter=filter_condition) except Exception as e: engLib.debug(f'Error deleting old chunks: {e}') + raise📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.# Erase all documents/chunks associated with that ObjectId in one operation # so we can cleanly insert the new version if len(objectIds.keys()): filter_condition = f"meta['objectId'] in [{', '.join(json.dumps(k) for k in objectIds.keys())}]" try: # Delete entities self.client.delete(collection_name=self.collection, filter=filter_condition) except Exception as e: engLib.debug(f'Error deleting old chunks: {e}') raise🧰 Tools
🪛 ast-grep (0.44.1)
[info] 472-472: use jsonify instead of json.dumps for JSON output
Context: json.dumps(k)
Note: [CWE-116] Improper Encoding or Escaping of Output.(use-jsonify)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_milvus/milvus.py` around lines 470 - 478, Update the delete error handling in addChunks so failures from self.client.delete are logged and then re-raised, matching the behavior in remove(). Do not continue inserting replacement chunks when deletion of existing objectId chunks fails; preserve the current successful deletion and insertion flow.
663-674: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
Critical:
render()reads a non-existent top-levelchunkIdfield — will raiseKeyErroron every non-empty result.The query only requests
output_fields=['meta', 'content'], and the collection schema (see_createCollection, lines 150-169) has no top-levelchunkIdfield —chunkIdlives inside the JSONmetablob.getPaths()correctly accesses it viarecord.get('meta', {}).get('chunkId'), but herechunk = point['chunkId']will raiseKeyErrorfor every returned point, breaking the entire "Rendering" feature described in the README whenever an object has stored chunks.Additionally, unlike every sibling store's
render()(chroma, qdrant, pinecone, elasticsearch — per the provided graph context), this loop is missing the upper-bound guard (if chunk >= offset + self.renderChunkSize: continue), so once the KeyError is fixed, an out-of-rangechunkIdfrom a stale/dirty query result could still cause anIndexErrorontext[index] = content.🐛 Proposed fix
for point in results: content = point['content'] - chunk = point['chunkId'] + meta = point.get('meta', {}) or {} + chunk = meta.get('chunkId') index = chunk - offset # Should never happen since we gave it an offset if chunk < offset: continue + # Should never happen since we gave it a range + if chunk >= offset + self.renderChunkSize: + continue + text[index] = content if index > lastIndex: lastIndex = index📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.for point in results: content = point['content'] meta = point.get('meta', {}) or {} chunk = meta.get('chunkId') index = chunk - offset # Should never happen since we gave it an offset if chunk < offset: continue # Should never happen since we gave it a range if chunk >= offset + self.renderChunkSize: continue text[index] = content if index > lastIndex: lastIndex = index🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_milvus/milvus.py` around lines 663 - 674, Update the result-processing loop in render() to read chunkId from each point’s meta JSON, matching getPaths(), instead of accessing a non-existent top-level field. Add the upper-bound guard so chunks at or beyond offset + self.renderChunkSize are skipped before indexing text, while preserving the existing lower-bound check and rendering behavior for valid chunks.nodes/src/nodes/vectordb_pinecone/IGlobal.py (1)
177-182: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
pinecone python SDK list_indexes return type IndexList index model .get name spec💡 Result:
In the Pinecone Python SDK, the list_indexes method returns an IndexList object [1][2][3]. This object is a wrapper around a collection of IndexModel instances and provides convenience methods for interacting with the returned data [4][5]. Key details regarding the return type and object structure include: Accessing Index Models: While the IndexList object is iterable and supports length checks (len) and direct indexing (e.g., indexes), it is a wrapper class [4][5]. To access the actual list of IndexModel objects, you should access the.indexes attribute (e.g., pc.list_indexes.indexes) [4][5][3]. Convenience Methods: -.names: Returns a list of strings containing the names of all indexes in the project [4][5]. -.to_dict: Returns a serializable dictionary representation of the index list, where each entry is converted to a dictionary [4][5]. IndexModel Structure: Each IndexModel instance (accessible via iteration or direct indexing on the IndexList) contains metadata about the index, including attributes such as: - name: The name of the index [6]. - spec: The specification of the index (e.g., serverless or pod-based configuration) [6][7]. - status: The current status of the index (e.g., ready state) [6][7]. - metric, dimension, etc.: Other relevant configuration details [6][7]. Example Usage: from pinecone import Pinecone pc = Pinecone(api_key='YOUR_API_KEY') indexes = pc.list_indexes # Iterate over index models for index in indexes.indexes: print(f"Name: {index.name}, Spec: {index.spec}") # Get list of names names = indexes.names # Access a specific model directly first_index = indexes.indexes
Citations:
- 1: https://sdk.pinecone.io/python/reference/pinecone.html
- 2: https://sdk.pinecone.io/python/_modules/pinecone/client/indexes.html
- 3: https://theneuralbase.com/pinecone/learn/beginner/verifying-with-pc-list-indexes/
- 4: https://sdk.pinecone.io/python/_modules/pinecone/models/indexes/list.html
- 5: https://sdk.pinecone.io/python/reference/models.html
- 6: https://sdk.pinecone.io/python/_modules/pinecone/models/indexes/index.html
- 7: https://docs.pinecone.io/reference/api/2025-04/control-plane/list_indexes
🌐 Web query:
Pinecone Python SDK IndexModel .get() spec attribute list_indexes iterable direct indexing💡 Result:
In the Pinecone Python SDK, IndexModel objects and the associated list_indexes response provide specific methods for accessing index details and iterating over collections. IndexModel and the spec attribute: The IndexModel class, which is returned by methods like pc.indexes.describe("index_name"), contains a spec attribute [1][2]. This attribute is an instance of IndexSpec, which holds deployment configuration (e.g., serverless, pod, or byoc) [1]. While you cannot use direct dictionary-style indexing (e.g., index_model['spec']) to access this, you can access it via dot notation (e.g., index_model.spec.serverless.cloud) [2]. To get a dictionary representation of the entire model, you can use the.to_dict method, which recursively converts the spec and other nested fields into a plain dictionary [1][3]. list_indexes and iteration: The pc.list_indexes method returns an IndexList object, which acts as a wrapper around a list of IndexModel instances [4]. This object is iterable, meaning you can loop over it directly to access each IndexModel [5][4]. For example: for index in pc.list_indexes: print(index.name) Additionally, the IndexList object provides a.names method if you only need the names of the indexes, which returns an iterable list of strings [5][4]. Direct Indexing: The SDK does not support direct indexing (dictionary-style lookup) on IndexModel or IndexList objects themselves [4]. Access must be performed through their defined attributes or helper methods like.to_dict [1][4]. To interact with a specific index's data plane, you use pc.index("index_name") to obtain an Index client instance [6][7].
Citations:
- 1: https://sdk.pinecone.io/python/reference/models.html
- 2: https://sdk.pinecone.io/python/how-to/indexes/serverless.html
- 3: https://sdk.pinecone.io/python/_modules/pinecone/models/indexes/index.html
- 4: https://sdk.pinecone.io/python/_modules/pinecone/models/indexes/list.html
- 5: https://sdk.pinecone.io/python/db_control/shared-index-actions.html
- 6: https://sdk.pinecone.io/python/reference/sync-index.html
- 7: https://sdk.pinecone.io/python/getting-started/quickstart.html
🏁 Script executed:
#!/bin/bash set -euo pipefail file='nodes/src/nodes/vectordb_pinecone/IGlobal.py' nl -ba "$file" | sed -n '160,195p' printf '\n---\n' rg -n "list_indexes|IndexModel|to_dict|spec.serverless|serverless" nodes/src/nodes/vectordb_pinecone -SRepository: rocketride-org/rocketride-server
Length of output: 210
🏁 Script executed:
#!/bin/bash set -euo pipefail file='nodes/src/nodes/vectordb_pinecone/IGlobal.py' awk 'NR>=160 && NR<=195 { printf "%d:%s\n", NR, $0 }' "$file" printf '\n---\n' rg -n "list_indexes|existing_collection|get\\('name'\\)|get\\('spec'\\)|serverless" "$file" -n -SRepository: rocketride-org/rocketride-server
Length of output: 2977
Use Pinecone model attributes here
client.list_indexes()yieldsIndexModelitems, soindex.get('name')andexisting_collection.get('spec', {})won’t work; useindex.name/existing_collection.spec(orto_dict()first) before checking mode compatibility.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_pinecone/IGlobal.py` around lines 177 - 182, Update the collection lookup and mode validation in the Pinecone initialization flow to use IndexModel attributes: compare `index.name` when finding `existing_collection`, then inspect `existing_collection.spec` for serverless mode instead of calling `.get()` on the model. Preserve the existing compatibility-check behavior.Source: Linters/SAST tools
nodes/src/nodes/vectordb_pinecone/pinecone.py (2)
238-245: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
searchKeyworddouble-converts the filter and passes a dict intoget().
get()unconditionally callsself._convertFilter(docFilter=docFilter), which accessesDocFilterattributes (.isDeleted,.nodeId, …). Herefilteris already a converted{'$and': [...]}dict, soget()will raiseAttributeErroron the dict, and the appended$containspredicate is discarded even if it didn't crash. Keyword search is broken on any invocation.Pass the
DocFilterthrough and inject the keyword predicate inside the query path, or add a raw-filter branch toget().🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_pinecone/pinecone.py` around lines 238 - 245, Update searchKeyword so it does not pass the already converted filter dict to get(), since get() unconditionally calls _convertFilter and expects a DocFilter. Preserve the keyword predicate by either passing the original DocFilter and applying the content condition within the query path, or adding an explicit raw-filter path to get() that supports the converted filter.
288-290: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Score threshold is read from the wrong (misspelled) attribute, so it never applies.
__init__setsself.threshold_search(Line 90), but this filter readsself.threshhold_search(the misspelled class attribute at Line 68, permanently0.0). Every non-negative score passes, so the configuredscorethreshold is silently ignored for semantic search.🐛 Proposed fix
- points_satisfy_threshold = [point for point in points if point['score'] >= self.threshhold_search] + points_satisfy_threshold = [point for point in points if point['score'] >= self.threshold_search]Also rename the class attribute at Line 68 (
threshhold_search→threshold_search) so the declared field matches the value set in__init__.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.# Have to get similarity score threshold post query points_satisfy_threshold = [point for point in points if point['score'] >= self.threshold_search]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_pinecone/pinecone.py` around lines 288 - 290, Fix the threshold attribute naming in the Pinecone node: rename the class attribute threshhold_search to threshold_search and update the points_satisfy_threshold filter to read self.threshold_search, matching the value initialized by __init__ so semantic search applies the configured score threshold.nodes/src/nodes/vectordb_pinecone/README.md (1)
54-59: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Default-profile documentation contradicts the schema.
This section marks
serverless-denseas the default ("The default profile is serverless-dense"), but the Fields table (Line 36) and the generated schema (Line 112) both give theprofiledefault as"pod-based". Reconcile so the docs state a single, correct default.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_pinecone/README.md` around lines 54 - 59, Reconcile the Pinecone profile default documentation with the schema by updating the conflicting default references in the README, including the profile table and the statement below it, so they consistently identify pod-based as the default.nodes/src/nodes/vectordb_qdrant/qdrant.py (1)
140-149: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value
Consider closing the client in
__del__.
__del__clears attributes but never callsself.client.close(), unlikeIGlobal.validateConfig()which explicitly closes its probe client. SinceQdrantClientholds open HTTP connections, closing it here would be more consistent and avoid relying on GC to release sockets.♻️ Proposed fix
def __del__(self): """ Deinitializes the qdrant client. """ # Deinit everything we did self.apikey = None self.collection = '' self.renderChunkSize = 0 self.similarity = 'Cosine' + if self.client is not None: + try: + self.client.close() + except Exception: + pass self.client = None📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def __del__(self): """ Deinitializes the qdrant client. """ # Deinit everything we did self.apikey = None self.collection = '' self.renderChunkSize = 0 self.similarity = 'Cosine' if self.client is not None: try: self.client.close() except Exception: pass self.client = None🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_qdrant/qdrant.py` around lines 140 - 149, Update Qdrant’s __del__ method to close the existing self.client before clearing it, while safely handling the client being unset or unavailable. Preserve the current attribute-reset behavior after cleanup.nodes/src/nodes/vectordb_weaviate/IGlobal.py (1)
71-154: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
validateConfiginfers cloud/local from the hostname instead of the configuredmode, which can diverge from whatStore.__init__actually connects to.
Store.__init__(weaviate.py) decides cloud vs. local viaconfig.get('mode'), butvalidateConfighere re-derives the same decision fromhoststring patterns (localhost/127.*). If a user picks the "local" profile with a non-loopback host (common with Docker/K8s service names or private IPs), or "cloud" pointed at a tunnel,validateConfigwill probe the wrong protocol/endpoint (skipping the gRPC readiness check for a real local connection, or vice versa), giving misleading save-time validation results relative to actual runtime behavior.🐛 Proposed fix
config = Config.getNodeConfig(self.glb.logicalType, self.glb.connConfig) host = (config.get('host', '')).strip() port = config.get('port') grpc_port = config.get('grpc_port') apikey = (config.get('apikey', '')).strip() collection = (config.get('collection', '')).strip() + profile = config.get('mode') @@ - # Decide cloud vs local: treat localhost/127.* as local; otherwise cloud - is_local = host in ('localhost', '127.0.0.1') or host.startswith('127.') - is_cloud = not is_local + # Match the same decision Store.__init__ uses at runtime + is_cloud = profile == 'cloud' + is_local = not is_cloud📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def validateConfig(self): """ Validate Weaviate config at save-time with a fast, SDK-driven probe. - Cloud: connect_to_weaviate_cloud + client.is_ready() - Local: connect_to_local (host, port, grpc_port) + client.is_ready() - Collection name format check (no existence check) - Minimal timeouts; SDK/HTTP exceptions surfaced without truncation """ try: # Get config config = Config.getNodeConfig(self.glb.logicalType, self.glb.connConfig) host = (config.get('host', '')).strip() port = config.get('port') grpc_port = config.get('grpc_port') apikey = (config.get('apikey', '')).strip() collection = (config.get('collection', '')).strip() profile = config.get('mode') # Collection name validation (official Weaviate rule) if not WEAVIATE_COLLECTION_RE.fullmatch(collection or ''): warning( 'Start with uppercase; only letters/numbers/underscore; no spaces or special characters (/ , . : ; \' " { } ( ) % ^ & $ # @ - ! ? * + = [ ] | \\ ~ < >).' ) return # Load dependencies before importing SDK from depends import depends # type: ignore requirements = os.path.dirname(os.path.realpath(__file__)) + '/requirements.txt' depends(requirements) # Imports used for SDK clients import weaviate # type: ignore from weaviate.classes.init import AdditionalConfig, Timeout # type: ignore import httpx # type: ignore import socket # type: ignore # Match the same decision Store.__init__ uses at runtime is_cloud = profile == 'cloud' is_local = not is_cloud # Build base URL (add scheme if missing) if is_local: base_url = host if host.startswith('http') else f'http://{host}' if port: base_url = f'{base_url}:{port}' else: base_url = host if host.startswith('https://') else f'https://{host}' if is_cloud: # Cloud: REST + API key only; do not probe gRPC url = base_url.rstrip('/') + '/v1/meta' headers = {'Authorization': f'Bearer {apikey}'} if apikey else {} r = httpx.get(url, headers=headers, timeout=3) r.raise_for_status() return else: # Local: use SDK client and a lightweight call (no API key) client = weaviate.connect_to_local( host=host, port=port, grpc_port=grpc_port, additional_config=AdditionalConfig(timeout=Timeout(init=3, query=3)), ) try: # REST check via SDK (hits /v1/schema) client.collections.list_all() # gRPC READY (grpc_port is required by UI) gp = int(grpc_port) try: import grpc # type: ignore grpc.channel_ready_future(grpc.insecure_channel(f'{host}:{gp}')).result(timeout=3) except ImportError: with socket.create_connection((host, gp), timeout=3): pass finally: try: client.close() except Exception: pass except Exception as e: warning(_format_error(e))🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 113-113: Do not make http calls without encryption
Context: f'http://{host}'
Note: [CWE-319] Cleartext Transmission of Sensitive Information.(requests-http)
[warning] 123-123: Request-controlled URL passed to httpx; validate against an allowlist to prevent SSRF.
Context: httpx.get(url, headers=headers, timeout=3)
Note: [CWE-918] Server-Side Request Forgery (SSRF).(avoid-ssrf)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_weaviate/IGlobal.py` around lines 71 - 154, Update validateConfig to determine cloud versus local using the configured mode value, matching the mode selection used by Store.__init__ in weaviate.py, instead of inferring it from host patterns. Preserve the existing cloud HTTP probe and local SDK/gRPC readiness checks, but route them according to the selected mode so save-time validation matches runtime behavior.nodes/src/nodes/vectordb_weaviate/weaviate.py (2)
489-541: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
Per-object update loop for
markDeleted/markActive— consider batching.Both methods issue one
collectionObj.data.update(...)call per matching point. For soft-delete/restore operations touching thousands of chunks (a plausible scenario per the object-level re-ingestion pattern documented in the README), this becomes an O(n) round-trip cost. Worth checking whether Weaviate's batch API (already used inaddChunks) supports partial-property batch updates to reduce this to a single batched call.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_weaviate/weaviate.py` around lines 489 - 541, Update markDeleted and markActive to use Weaviate’s batch API, consistent with addChunks, instead of calling collectionObj.data.update once per point. Batch the matching point UUIDs with the appropriate isDeleted value while preserving the existing collection checks, empty-input handling, and filtering behavior.
543-606: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail # Inspect the relevant file sections and nearby query usages python3 - <<'PY' from pathlib import Path path = Path('nodes/src/nodes/vectordb_weaviate/weaviate.py') lines = path.read_text().splitlines() for start, end in [(300, 420), (543, 606)]: print(f"\n--- {path} lines {start}-{end} ---") for i in range(start, min(end, len(lines)) + 1): print(f"{i:4d}: {lines[i-1]}") PY # Search for fetch_objects call sites in this file rg -n "fetch_objects\(" nodes/src/nodes/vectordb_weaviate/weaviate.pyRepository: rocketride-org/rocketride-server
Length of output: 8445
render()needs an explicitlimitonfetch_objectsWithout
limit=self.renderChunkSize, Weaviate falls back to its default page size (10), so any document with more than 10 chunks in a window is rendered incompletely and the remaining chunks are dropped.🐛 Proposed fix
points = self.collectionObj.query.fetch_objects( filters=( Filter.by_property('objectId').equal(objectId) & Filter.by_property('chunkId').greater_or_equal(offset) & Filter.by_property('chunkId').less_than(offset + self.renderChunkSize) - ) + ), + limit=self.renderChunkSize, )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def render(self, objectId: str, callback: Callable[[str], None]) -> None: """ Given an object id, render the complete document. Rehydrates all the chunks into the proper order. """ # By definition, if the collection does not exists, there # is nothing to render if not self.doesCollectionExist(): return # Since chunks are returned in any order, and a single objectId # may contain tens of thousands of chunks, we grave them one # group at a time (renderChunkSize), put them into an array, # join them and call the callback offset = 0 while True: # Build filter for getting a set of chunks within the offset range points = self.collectionObj.query.fetch_objects( filters=( Filter.by_property('objectId').equal(objectId) & Filter.by_property('chunkId').greater_or_equal(offset) & Filter.by_property('chunkId').less_than(offset + self.renderChunkSize) ), limit=self.renderChunkSize, ) # Create a renderChunkSize array with empty # entries. This will allow us to join even when # a chunk doesn't come back # Process the results text = [''] * self.renderChunkSize lastIndex = -1 for point in points.objects: content = point.properties.get('content') chunk = point.properties.get('chunkId') index = chunk - offset # Should never happen since we gave it an offset if chunk < offset: continue text[index] = content if index > lastIndex: lastIndex = index # Compute the number of items we are going to process numberOfItems = lastIndex + 1 # If we got no items back, we are done if numberOfItems < 1: break # Join it together fullText = ''.join(text[0:numberOfItems]) # Call the output function callback(fullText) # If we got less than we asked for, must be done if numberOfItems < self.renderChunkSize: break offset += self.renderChunkSize🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/vectordb_weaviate/weaviate.py` around lines 543 - 606, Update the fetch_objects call inside render to pass limit=self.renderChunkSize, ensuring each chunk window requests the full configured batch rather than Weaviate’s default page size. Preserve the existing offset filtering, ordering, callback behavior, and termination logic.nodes/test/vectordb_astra/test_convert_filter.py (1)
15-15: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
Stale import path:
nodes.astra_db.astra_dbno longer exists after the rename.This PR relocates the module to
nodes/src/nodes/vectordb_astra/, but this test still importsStorefromnodes.astra_db.astra_db. That package path should no longer resolve once the folder is renamed, so this test will fail withModuleNotFoundError/ImportErrorrather than exercising_convertFilter. This also contradicts the AI summary's claim that test import paths were updated for this cohort.🐛 Proposed fix
-from nodes.astra_db.astra_db import Store +from nodes.vectordb_astra.astra_db import Store#!/bin/bash # Confirm the module was moved and no `nodes.astra_db` package remains, and check other astra tests for the correct import path. fd -a astra_db.py rg -n 'nodes\.astra_db|nodes\.vectordb_astra' nodes/test -g '*.py'🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/test/vectordb_astra/test_convert_filter.py` at line 15, Update the Store import in test_convert_filter.py to use the relocated nodes.vectordb_astra module path, matching the package structure used by the moved implementation. Remove the stale nodes.astra_db.astra_db reference so the test can import Store and exercise _convertFilter.nodes/test/vectordb_milvus/test_addchunks_return.py (1)
19-23: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Stale
nodes.milvus.milvusimport inside the skipped test.This file was moved under
nodes/test/vectordb_milvus/, but the import inside the test body still targets the pre-rename module path. Since the test is skipped, this won't fail CI now, but it will break withModuleNotFoundErroras soon as the skip is removed.🐛 Proposed fix
- from nodes.milvus.milvus import Store + from nodes.vectordb_milvus.milvus import Store📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.`@pytest.mark.skip`(reason='todo: refactor this unit test') def test_milvus_addchunks_returns_none_when_create_collection_fails(): """``addChunks`` must return None when ``createCollection`` fails.""" from nodes.vectordb_milvus.milvus import Store🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/test/vectordb_milvus/test_addchunks_return.py` around lines 19 - 23, Update the import inside test_milvus_addchunks_returns_none_when_create_collection_fails to reference the current Milvus module path after the package move, removing the stale nodes.milvus.milvus target so the test can run when its skip is removed.
229a6c0 to
05ffd08
Compare
There was a problem hiding this comment.
Pull request overview
Renames the vector-store node family to the vectordb_* prefix and the graph-first database nodes to graph_*, updating service registrations, documentation redirects, and tests to match the new module paths while keeping runtime resolution stable via unchanged protocol/prefix.
Changes:
- Updated node service registrations (
path, and for graph nodes alsoclassType) to the newvectordb_*/graph_*module names. - Renamed/added node docs and dependency manifests under the new node directories.
- Updated tests and docs redirects so imports and doc URLs continue to resolve after the renames.
Reviewed changes
Copilot reviewed 32 out of 115 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| packages/docs/redirects.ts | Adds redirects from old /nodes/<old> doc routes to the new vectordb_* / graph_* routes. |
| nodes/test/weaviate/test_record_updates.py | Updates direct module-path loading to vectordb_weaviate. |
| nodes/test/tool_xtrace_memory/test_tools.py | Scopes engine-dependency stubs to import-time and restores sys.modules afterward. |
| nodes/test/tool_mem0/test_tools.py | Scopes engine-dependency stubs to import-time and restores sys.modules afterward. |
| nodes/test/tool_filesystem/test_read_size_cap.py | Scopes rocketlib stub to import-time and restores it afterward. |
| nodes/test/test_db_hydradb.py | Updates node directory reference to graph_hydradb. |
| nodes/test/test_db_arango.py | Updates node directory reference to graph_arango. |
| nodes/test/test_currency_convert_explicit.py | Scopes rocketlib stub to import-time and restores it afterward. |
| nodes/test/test_anomaly_detector.py | Scopes rocketlib stub to import-time and restores it afterward. |
| nodes/test/qdrant/test_search_threshold.py | Updates direct module load path to vectordb_qdrant. |
| nodes/test/qdrant/test_mark_deleted.py | Updates direct module load path to vectordb_qdrant. |
| nodes/test/pinecone/test_record_updates.py | Updates direct module load path to vectordb_pinecone. |
| nodes/test/ocr/test_model_server_ocr.py | Updates intra-test references to renamed test modules. |
| nodes/test/milvus/test_search_semantic.py | Updates sys.path insertion to vectordb_milvus. |
| nodes/test/milvus/test_filter_injection.py | Updates sys.path insertion to vectordb_milvus. |
| nodes/test/guardrails/test_all.py | Scopes rocketlib stub to import-time and restores it afterward. |
| nodes/test/conftest.py | Updates skip-list entry from index_search to vectordb_elasticsearch. |
| nodes/test/cloud_tts/test_cloud_tts.py | Scopes core stubs to import-time and restores sys.modules afterward. |
| nodes/test/chroma/test_convert_filter_explicit_none.py | Updates direct module load path to vectordb_chroma. |
| nodes/test/chroma/test_coerce_port.py | Updates direct module load path to vectordb_chroma. |
| nodes/test/atlas/test_isdeleted_filter.py | Updates direct module load path to vectordb_atlas and scopes pydantic stub. |
| nodes/src/nodes/vectordb_weaviate/services.json | Updates Python module path to nodes.vectordb_weaviate. |
| nodes/src/nodes/vectordb_weaviate/requirements.txt | Adds node dependency list under the renamed node directory. |
| nodes/src/nodes/vectordb_weaviate/README.md | Adds co-located node documentation under the renamed node directory. |
| nodes/src/nodes/vectordb_weaviate/IInstance.py | Adds node instance transform under the renamed node directory. |
| nodes/src/nodes/vectordb_weaviate/IGlobal.py | Adds node global transform + save-time config validation under the renamed node directory. |
| nodes/src/nodes/vectordb_weaviate/IEndpoint.py | Adds endpoint transform under the renamed node directory. |
| nodes/src/nodes/vectordb_weaviate/init.py | Adds node module exports under the renamed node directory. |
| nodes/src/nodes/vectordb_qdrant/services.json | Updates Python module path to nodes.vectordb_qdrant. |
| nodes/src/nodes/vectordb_qdrant/requirements.txt | Adds node dependency list under the renamed node directory. |
| nodes/src/nodes/vectordb_qdrant/README.md | Adds co-located node documentation under the renamed node directory. |
| nodes/src/nodes/vectordb_qdrant/qdrant.svg | Adds node icon under the renamed node directory. |
| nodes/src/nodes/vectordb_qdrant/IInstance.py | Adds node instance transform under the renamed node directory. |
| nodes/src/nodes/vectordb_qdrant/IGlobal.py | Adds node global transform + save-time config validation under the renamed node directory. |
| nodes/src/nodes/vectordb_qdrant/IEndpoint.py | Adds endpoint transform under the renamed node directory. |
| nodes/src/nodes/vectordb_qdrant/init.py | Adds node module exports under the renamed node directory. |
| nodes/src/nodes/vectordb_pinecone/services.json | Updates Python module path to nodes.vectordb_pinecone. |
| nodes/src/nodes/vectordb_pinecone/requirements.txt | Adds node dependency list under the renamed node directory. |
| nodes/src/nodes/vectordb_pinecone/README.md | Adds co-located node documentation under the renamed node directory. |
| nodes/src/nodes/vectordb_pinecone/pinecone.svg | Adds node icon under the renamed node directory. |
| nodes/src/nodes/vectordb_pinecone/IInstance.py | Adds node instance transform under the renamed node directory. |
| nodes/src/nodes/vectordb_pinecone/IGlobal.py | Adds node global transform + save-time config validation under the renamed node directory. |
| nodes/src/nodes/vectordb_pinecone/IEndpoint.py | Adds endpoint transform under the renamed node directory. |
| nodes/src/nodes/vectordb_pinecone/init.py | Adds node module exports under the renamed node directory. |
| nodes/src/nodes/vectordb_milvus/services.json | Updates Python module path to nodes.vectordb_milvus. |
| nodes/src/nodes/vectordb_milvus/requirements.txt | Adds node dependency list under the renamed node directory. |
| nodes/src/nodes/vectordb_milvus/README.md | Adds co-located node documentation under the renamed node directory. |
| nodes/src/nodes/vectordb_milvus/IInstance.py | Adds node instance transform under the renamed node directory. |
| nodes/src/nodes/vectordb_milvus/IGlobal.py | Adds node global transform + save-time config validation under the renamed node directory. |
| nodes/src/nodes/vectordb_milvus/IEndpoint.py | Adds endpoint transform under the renamed node directory. |
| nodes/src/nodes/vectordb_milvus/init.py | Adds node module exports under the renamed node directory. |
| nodes/src/nodes/vectordb_elasticsearch/services.opensearch.json | Updates Python module path to nodes.vectordb_elasticsearch for OpenSearch service registration. |
| nodes/src/nodes/vectordb_elasticsearch/services.elasticsearch.json | Updates Python module path to nodes.vectordb_elasticsearch for Elasticsearch service registration. |
| nodes/src/nodes/vectordb_elasticsearch/requirements.txt | Adds combined Elasticsearch/OpenSearch dependency list under the renamed node directory. |
| nodes/src/nodes/vectordb_elasticsearch/opensearch/README.md | Adds OpenSearch docs page under the renamed node directory. |
| nodes/src/nodes/vectordb_elasticsearch/opensearch.svg | Adds OpenSearch icon under the renamed node directory. |
| nodes/src/nodes/vectordb_elasticsearch/IEndpoint.py | Adds endpoint transform under the renamed node directory. |
| nodes/src/nodes/vectordb_elasticsearch/elasticsearch/README.md | Adds Elasticsearch docs page under the renamed node directory. |
| nodes/src/nodes/vectordb_elasticsearch/elasticsearch.svg | Adds Elasticsearch icon under the renamed node directory. |
| nodes/src/nodes/vectordb_elasticsearch/constants.py | Introduces shared constants for the combined Elasticsearch/OpenSearch node. |
| nodes/src/nodes/vectordb_elasticsearch/init.py | Adds node module exports under the renamed node directory. |
| nodes/src/nodes/vectordb_chroma/services.json | Updates Python module path to nodes.vectordb_chroma. |
| nodes/src/nodes/vectordb_chroma/requirements.txt | Adds node dependency list under the renamed node directory. |
| nodes/src/nodes/vectordb_chroma/README.md | Adds co-located node documentation under the renamed node directory. |
| nodes/src/nodes/vectordb_chroma/IInstance.py | Adds node instance transform under the renamed node directory. |
| nodes/src/nodes/vectordb_chroma/IGlobal.py | Adds node global transform under the renamed node directory. |
| nodes/src/nodes/vectordb_chroma/IEndpoint.py | Adds endpoint transform under the renamed node directory. |
| nodes/src/nodes/vectordb_chroma/chroma.svg | Adds node icon under the renamed node directory. |
| nodes/src/nodes/vectordb_chroma/init.py | Adds node module exports under the renamed node directory. |
| nodes/src/nodes/vectordb_atlas/services.json | Updates Python module path to nodes.vectordb_atlas. |
| nodes/src/nodes/vectordb_atlas/requirements.txt | Adds node dependency list under the renamed node directory. |
| nodes/src/nodes/vectordb_atlas/README.md | Adds co-located node documentation under the renamed node directory. |
| nodes/src/nodes/vectordb_atlas/mongodb.svg | Adds node icon under the renamed node directory. |
| nodes/src/nodes/vectordb_atlas/IInstance.py | Adds node instance transform under the renamed node directory. |
| nodes/src/nodes/vectordb_atlas/IGlobal.py | Adds node global transform + save-time config validation under the renamed node directory. |
| nodes/src/nodes/vectordb_atlas/IEndpoint.py | Adds endpoint transform under the renamed node directory. |
| nodes/src/nodes/vectordb_atlas/init.py | Adds node module exports under the renamed node directory. |
| nodes/src/nodes/vectordb_astra/services.json | Updates Python module path to nodes.vectordb_astra. |
| nodes/src/nodes/vectordb_astra/requirements.txt | Adds node dependency list under the renamed node directory. |
| nodes/src/nodes/vectordb_astra/README.md | Adds co-located node documentation under the renamed node directory. |
| nodes/src/nodes/vectordb_astra/IInstance.py | Adds node instance transform under the renamed node directory. |
| nodes/src/nodes/vectordb_astra/IGlobal.py | Adds node global transform under the renamed node directory. |
| nodes/src/nodes/vectordb_astra/IEndpoint.py | Adds endpoint transform under the renamed node directory. |
| nodes/src/nodes/vectordb_astra/astra_db.svg | Adds node icon under the renamed node directory. |
| nodes/src/nodes/vectordb_astra/init.py | Adds node module exports under the renamed node directory. |
| nodes/src/nodes/graph_hydradb/services.json | Adds graph to classType and updates Python module path to nodes.graph_hydradb. |
| nodes/src/nodes/graph_hydradb/requirements.txt | Adds node dependency list under the renamed node directory. |
| nodes/src/nodes/graph_hydradb/README.md | Adds co-located node documentation under the renamed node directory. |
| nodes/src/nodes/graph_hydradb/IInstance.py | Adds tool-based HydraDB instance implementation under the renamed node directory. |
| nodes/src/nodes/graph_hydradb/IGlobal.py | Adds HydraDB global config/client wiring under the renamed node directory. |
| nodes/src/nodes/graph_hydradb/hydradb.svg | Adds node icon under the renamed node directory. |
| nodes/src/nodes/graph_hydradb/hydradb_client.py | Adds a thin HydraDB v2 REST client wrapper for the node tools. |
| nodes/src/nodes/graph_hydradb/init.py | Bootstraps runtime dependency loading and exports node classes. |
| nodes/src/nodes/graph_arango/utils.py | Adds shared utilities under the renamed node directory. |
| nodes/src/nodes/graph_arango/services.json | Adds graph to classType and updates Python module path to nodes.graph_arango. |
| nodes/src/nodes/graph_arango/requirements.txt | Adds node dependency list under the renamed node directory. |
| nodes/src/nodes/graph_arango/README.md | Adds co-located node documentation under the renamed node directory. |
| nodes/src/nodes/graph_arango/arango.svg | Adds node icon under the renamed node directory. |
| nodes/src/nodes/graph_arango/init.py | Bootstraps driver installation and exports node classes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
3de65d5 to
29a3dc3
Compare
29a3dc3 to
41a3274
Compare
|
Heads up on CI here: this PR is renames-only and depends on #1642 (test isolation, fixes #1640). CI on this branch stays red until #1642 lands on Validated end-to-end: applying #1642's fixes on top of this branch turned all three OS builds green (Ubuntu/macOS/Windows). Merge order: #1642 → develop, then rebase this. |
41a3274 to
bfa71f8
Compare
…o graph_* Rename the nine vector-store nodes to the store_ prefix (matching their `store` classType) and the two graph-first databases (ArangoDB, HydraDB) to graph_. Update each services.json path to the new python module and add graph to the ArangoDB/HydraDB classType (database and tool kept). protocol and prefix are left untouched, so saved pipes keep resolving; no services-version bump. Docs redirects map the old /nodes/<old> URLs to the new ones. Test module imports and path references follow the renamed src folders. constraints.lock and docs static pages are regenerated by CI. Node test folders keep their original names so pytest collection order is unchanged.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 30 out of 118 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
packages/docs/redirects.ts:120
- PR description says the vector-store nodes are being renamed to the “vectordb_” prefix, but the actual redirects (and node folders/paths in this PR) use the “store_” prefix (e.g. /nodes/store_astra). Please update the PR description to match the implemented naming so readers aren’t misled about the intended convention.
There was a problem hiding this comment.
Actionable comments posted: 51
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@nodes/src/nodes/graph_arango/IGlobal.py`:
- Around line 184-204: Update _run_query_raw so affected_rows reflects writes
even when the query returns rows: use len(rows) as the affected count when rows
are present, while retaining the cursor-derived affected count when no rows are
returned. Keep the existing row collection and max_execute_rows enforcement
unchanged.
In `@nodes/src/nodes/graph_arango/IInstance.py`:
- Around line 175-224: Update the get_aql tool’s input_schema to document the
supported limit parameter alongside question, including its expected numeric
type and purpose, while preserving the existing _clamp_limit(args.get('limit'))
behavior.
In `@nodes/src/nodes/graph_arango/README.md`:
- Line 1: Update the README H1 and its “View source” link to use the renamed
graph_arango node and path, replacing the stale db_arango references while
preserving the existing link structure.
In `@nodes/src/nodes/store_astra/astra_db.py`:
- Around line 490-529: Update Store.render to process object chunks in batches
using the store’s renderChunkSize configuration, rather than collecting,
sorting, joining, and callbacking the entire object at once. Preserve chunk
ordering by fetching each batch in chunkId order, invoke callback once per
batch, and add or reuse the appropriate renderChunkSize attribute consistent
with sibling stores.
- Around line 318-357: Update getPaths to exclude soft-deleted records by adding
the standard meta.isDeleted: False condition to its aggregation match filter,
while preserving the existing parent filter, pagination, and path-building
behavior.
- Around line 276-299: Update searchSemantic to raise a clear exception when
query.embedding is None and reject any non-zero docFilter.offset before querying
Astra. Preserve the existing collection check and semantic search flow for valid
embeddings with offset zero.
- Around line 147-200: Escape the user-supplied parent path before assigning it
to the $regex filter in _convertFilter. Use the standard regex-escaping utility
so metacharacters in docFilter.parent are matched literally, while preserving
the existing parent filter behavior and all other filter clauses.
- Around line 358-435: Update the near-zero-vector validation inside
addChunks.flush so the schema control document created by the zero-embedding
workaround is retained for insertion. Ensure the validity check or
schema-specific handling accepts that boosted control vector while continuing to
filter genuinely invalid vectors for other documents, preserving
doesCollectionExist control-document detection.
- Around line 62-90: Update api_endpoint initialization in the store setup to
require a configured value, normalize it by trimming whitespace, removing the
http/https scheme and surrounding trailing slashes as described by the existing
comment, and validate the resulting endpoint before passing it to get_database.
Raise a clear configuration error for missing or invalid endpoints, while
preserving the existing collection_name, similarity, and application_token
handling.
In `@nodes/src/nodes/store_astra/README.md`:
- Around line 84-99: Regenerate the documentation for the renamed store_astra
node using the nodes:docs-generate workflow so the generated Source link points
to nodes/src/nodes/store_astra instead of the obsolete astra_db path. Do not
manually edit the generated block.
In `@nodes/src/nodes/store_atlas/atlas.py`:
- Around line 351-352: Replace the shallow sys.getsizeof(doc) calculation in the
document batching loop with an estimate of the serialized payload that includes
content bytes and embedding element storage, or BSON-encode the document for an
exact measure. Keep accumulating the estimate in sum_size so the existing
payload_limit flush condition reflects actual document size.
- Around line 318-353: Update addChunks to validate that every chunk has a
non-None embedding before computing and deleting existing object IDs. Move the
embedding validation ahead of the objectIds deletion block, while preserving the
existing exception behavior and document preparation flow.
- Around line 270-276: Initialize results to a safe empty collection before the
try block, or return an appropriate empty-document result directly from the
OperationFailure handler, so _convertToDocs receives a defined value when
aggregate fails. Preserve the existing successful aggregation and logging
behavior.
- Around line 236-249: Remove the redundant model-parameter collection check and
its misleading validation comment from searchSemantic. Keep the initial
doesCollectionExist() guard returning [] for missing collections, while
preserving the separate embedding and offset validations.
- Around line 294-316: Escape externally supplied parent filter values before
constructing MongoDB `$regex` conditions in both `getPaths()` and
`_convertFilter()`. Apply `re.escape(...)` to `parent` and `docFilter.parent`
while preserving the existing case-insensitive matching behavior and filter
structure.
In `@nodes/src/nodes/store_atlas/README.md`:
- Around line 74-77: Regenerate the documentation for the renamed store_atlas
node using nodes:docs-generate so the generated “View source” link targets
nodes/src/nodes/store_atlas. Do not manually edit content within the
ROCKETRIDE:GENERATED:PARAMS block.
In `@nodes/src/nodes/store_chroma/chroma.py`:
- Around line 245-253: Update the cosine branch in the score calculation around
self.similarity to convert Chroma’s distance into a similarity score using 1 -
distance / 2, so lower distances produce higher scores. Apply
self.threshold_search to this normalized score instead of the hardcoded 0.20
threshold, while preserving the existing non-cosine scoring path.
- Around line 498-503: The soft-delete and reactivation flows in markDeleted and
markActive incorrectly iterate Chroma result keys and call update with
unsupported arguments. At both affected sites in
nodes/src/nodes/store_chroma/chroma.py (lines 498-503 and 522-527), build
parallel ids and metadatas lists from the returned result columns, update each
metadata object’s isDeleted value while preserving all existing keys, then call
self.collectionObj.update with ids and metadatas.
- Around line 374-378: Update the getPaths filter construction to use the
chunkId metadata key with an equality condition for 0. When parent is provided,
combine the chunkId and parent conditions under a $and wrapper, preserving the
existing parent serialization.
- Around line 567-591: Update the collection.get call in render to request both
documents and metadatas, then read each row’s chunkId from the returned metadata
dictionary before applying the offset and range checks. Preserve the existing
text-array population logic and avoid treating metadata as an object attribute.
In `@nodes/src/nodes/store_elasticsearch/elasticsearch_store.py`:
- Around line 583-591: The payload batching logic in the bulk document loop must
measure serialized document content rather than the shallow dict size from
sys.getsizeof(doc). Update the size calculation near _run_bulk to use the
serialized _source (or equivalent payload-size estimate), and reuse
DEFAULT_TEXT_BATCH_SIZE instead of the hardcoded 500 when appropriate; preserve
flushing when either count or payload limits are reached.
- Around line 382-383: Update the offset validation in the Elasticsearch store’s
semantic-search flow to raise Exception instead of BaseException, preserving the
existing error message and condition for non-zero docFilter.offset values.
- Around line 93-116: Update the Elasticsearch configuration mapping before
Store.__init__ so deployment mode is resolved from the Elasticsearch
deployment/profile field, not config.mode. Pass the resolved profile-based
scheme or mode into Store, and ensure Store.__init__ uses that deployment value
when selecting the default URL scheme while preserving explicit http/https host
schemes.
In `@nodes/src/nodes/store_elasticsearch/README.md`:
- Line 187: Update the source metadata used by the README generator for the
Elasticsearch node so its generated source link targets
nodes/src/nodes/store_elasticsearch instead of nodes/src/nodes/index_search.
Then rerun nodes:docs-generate to regenerate the README block; do not edit the
generated link manually.
- Around line 37-75: Replace the outdated Elasticsearch configuration table with
entries matching the services.elasticsearch.json schema and Elasticsearch
loader: use the profile/vector fields shown in the lower table, remove
OpenSearch-only keys such as host, enabled, username, password, and collection,
and set provider to "elasticsearch" where applicable. Keep the documented
defaults and profile-specific fields accurate.
- Around line 124-126: Update the OpenSearch “Basic auth” documentation to
describe verify_certs=False and ssl_show_warn=False as an insecure compatibility
fallback, not suitable for production. Explicitly instruct production users with
self-signed certificates to configure CA certificate verification before
deployment, while preserving the existing authentication requirements.
In `@nodes/src/nodes/store_milvus/IGlobal.py`:
- Around line 74-77: Update the cloud validation branch in IGlobal.py to
construct the URI with the same https:// scheme used by the runtime Milvus
client before calling connections.connect. Preserve the raw host and port values
while ensuring token-based cloud validation receives a valid
http[s]://host[:port] URI.
In `@nodes/src/nodes/store_milvus/milvus.py`:
- Around line 417-431: Replace the json.dumps-based escaping in getPaths and the
delete filter within addChunks with the existing _escape_milvus_str helper.
Preserve the current filter semantics while routing both parent values through
the shared, tested escaping mechanism.
- Around line 450-520: In addChunks, validate that every chunk has a non-None
embedding before building objectIds or executing the existing deletion via
self.client.delete. Move the embedding check out of the insertion loop into a
preliminary validation pass, while preserving the current exception behavior and
upsert flow for valid batches.
- Around line 663-666: Update render() to read chunkId from the result’s meta
JSON/object, matching the existing getPaths() and _convertToDocs() handling,
instead of accessing point['chunkId'] directly. Preserve the current
chunk-offset calculation and rendering behavior after extracting the nested
value.
- Around line 104-109: Update the similarity validation in the Milvus
initialization logic to accept only COSINE, L2, and IP for the FLOAT_VECTOR
schema. Remove JACCARD, HAMMING, and BM25 from the allowed values while
preserving the existing exception for unsupported metrics.
- Around line 205-207: Update the docFilter.isTable condition to append a Milvus
filter using lowercase, unquoted boolean literals derived from the boolean
value, rather than _escape_milvus_str. Remove the extraneous tuple wrapper
around must_conditions.append so the condition is appended directly.
- Around line 144-193: The _createCollection method currently reverses the
expected success contract. Update its return values so an existing collection
and successful create_collection call return True, while exceptions during
creation return False; preserve the existing collection-existence check and
deadlock-avoidance behavior.
- Line 156: Remove the max_length argument from the INT64 primary-key field
definition in schema.add_field, while preserving the existing field_name,
datatype, and is_primary settings.
In `@nodes/src/nodes/store_pinecone/pinecone.py`:
- Line 68: Correct the misspelled class attribute threshhold_search to
threshold_search and update the searchSemantic filtering reference accordingly,
matching the self.threshold_search assignment in __init__. Ensure semantic
search uses the configured score threshold instead of the class-level default.
- Around line 226-249: Update searchKeyword and the related get flow so the
already-converted Pinecone filter is not passed back through _convertFilter as a
DocFilter. Separate filter conversion from filter application, or otherwise
reuse the converted filter directly, while preserving the existing DocFilter
conversion behavior for normal get callers.
- Around line 399-468: Update addChunks so flush only upserts pending points and
does not call self.remove or clear objectIds between batches. After the loop’s
final flush succeeds, delete the collected object IDs once, ensuring all new
chunks are written before stale data is removed and failures leave existing data
intact.
In `@nodes/src/nodes/store_pinecone/README.md`:
- Around line 52-59: Update the Profiles table and its concluding
default-profile statement in the README to use pod-based as the default,
matching the Fields table and generated Schema; mark serverless-dense as
non-default and pod-based as default without changing their deployment
descriptions.
In `@nodes/src/nodes/store_postgres/postgres.py`:
- Around line 408-478: Update the write methods _createCollection, addChunks,
remove, markDeleted, and markActive to wrap their cursor operations and commit
in try/except blocks; on any exception, call self.client.rollback() before
re-raising the original exception. Preserve existing behavior on successful
commits and early returns.
- Around line 424-439: The chunk insertion flow using self.client.cursor and
SQL_QUERIES['insert_chunk'] should batch all chunks into one database statement
instead of executing one INSERT per iteration. Build the complete row values and
shared column list first, then use psycopg2.extras.execute_values or an
equivalent multi-row INSERT, preserving vectorSize and modelName defaults before
committing once.
- Around line 123-127: Update the production connection in the store-postgres
constructor to pass psycopg2’s connect_timeout parameter, matching the 3-second
timeout used by IGlobal.validateConfig(). Keep the existing connection arguments
and register_vector(self.client) behavior unchanged.
In `@nodes/src/nodes/store_weaviate/IGlobal.py`:
- Around line 108-119: Update validateConfig’s cloud/local classification to
derive is_cloud from config.get('mode') == 'cloud', matching Store.__init__ in
weaviate.py, and derive is_local as its inverse. Remove the hostname-based
localhost/127.* heuristic while preserving the existing URL construction for
each resulting connection mode.
In `@nodes/src/nodes/store_weaviate/README.md`:
- Around line 126-129: Regenerate the documentation for the renamed
store_weaviate node using the nodes:docs-generate workflow so the generated
“View source” link targets nodes/src/nodes/store_weaviate instead of the
obsolete nodes/src/nodes/weaviate path. Do not edit the content between the
generated params markers manually.
In `@nodes/src/nodes/store_weaviate/requirements.txt`:
- Line 1: Update the authlib dependency declaration in requirements.txt to pin
it to version 1.7.2, matching the existing authlib==1.7.2 constraint in
constraints.lock.
In `@nodes/src/nodes/store_weaviate/weaviate.py`:
- Around line 415-469: Update addChunks to avoid deleting existing chunks before
the replacement batch has been imported successfully. Use an atomic or
rollback-safe replacement strategy—prefer UUID-based upsert when available,
otherwise insert the new chunks first and delete prior chunks only after the
batch completes without failed_objects—while preserving the existing objectId
replacement semantics.
- Around line 543-606: Update render() to replace the fixed-size text list
allocation with a sparse dictionary keyed by each point’s chunk - offset, so
memory usage scales with returned chunks rather than renderChunkSize. Preserve
chunk ordering, missing-chunk handling, lastIndex/numberOfItems calculation,
joined output, pagination, and callback behavior.
- Around line 489-514: Update markDeleted and markActive so their fetch_objects
queries do not rely on the default 10-result limit. Paginate through matching
objects in windows until no results remain, applying the existing isDeleted
update to every returned object. Preserve the collection-existence and
empty-objectIds early returns.
- Around line 267-277: Update the cosine branch in the score calculation around
self.similarity to normalize Weaviate distances using certainty semantics:
compute the score as 1 minus distance divided by 2. Keep the existing non-cosine
calculation, threshold check, and missing-distance behavior unchanged.
- Around line 302-326: The searchKeyword method currently interpolates the
entire QuestionText model into the Weaviate content filter. Update the filter
construction to use query.text so the wildcard pattern matches the question text
directly.
In `@nodes/test/conftest.py`:
- Around line 320-321: Track the temporary exclusion of store_elasticsearch for
re-enablement by adding a follow-up issue reference or TODO next to its entry in
the failing-test exclusion group. Keep the exclusion unchanged until dependency
`#1642` is resolved, and make the tracking explicit so the dynamically generated
tests can be restored afterward.
In `@packages/docs/redirects.ts`:
- Around line 183-197: Update the redirect entries in the redirects
configuration so /nodes/index_search continues targeting
/nodes/store_elasticsearch, while Elasticsearch-specific legacy paths target the
nested Elasticsearch page and OpenSearch-specific paths target the nested
OpenSearch page. Preserve the existing source paths and split the
provider-specific mappings accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: c38bf400-e060-48af-bae1-7cb4d95155ae
⛔ Files ignored due to path filters (12)
nodes/src/nodes/graph_arango/arango.svgis excluded by!**/*.svgnodes/src/nodes/graph_hydradb/hydradb.svgis excluded by!**/*.svgnodes/src/nodes/store_astra/astra_db.svgis excluded by!**/*.svgnodes/src/nodes/store_atlas/mongodb.svgis excluded by!**/*.svgnodes/src/nodes/store_chroma/chroma.svgis excluded by!**/*.svgnodes/src/nodes/store_elasticsearch/elasticsearch.svgis excluded by!**/*.svgnodes/src/nodes/store_elasticsearch/opensearch.svgis excluded by!**/*.svgnodes/src/nodes/store_milvus/milvus.svgis excluded by!**/*.svgnodes/src/nodes/store_pinecone/pinecone.svgis excluded by!**/*.svgnodes/src/nodes/store_postgres/postgres.svgis excluded by!**/*.svgnodes/src/nodes/store_qdrant/qdrant.svgis excluded by!**/*.svgnodes/src/nodes/store_weaviate/weaviate.svgis excluded by!**/*.svg
📒 Files selected for processing (106)
nodes/src/nodes/graph_arango/IGlobal.pynodes/src/nodes/graph_arango/IInstance.pynodes/src/nodes/graph_arango/README.mdnodes/src/nodes/graph_arango/__init__.pynodes/src/nodes/graph_arango/requirements.txtnodes/src/nodes/graph_arango/services.jsonnodes/src/nodes/graph_arango/utils.pynodes/src/nodes/graph_hydradb/IGlobal.pynodes/src/nodes/graph_hydradb/IInstance.pynodes/src/nodes/graph_hydradb/README.mdnodes/src/nodes/graph_hydradb/__init__.pynodes/src/nodes/graph_hydradb/hydradb_client.pynodes/src/nodes/graph_hydradb/requirements.txtnodes/src/nodes/graph_hydradb/services.jsonnodes/src/nodes/store_astra/IEndpoint.pynodes/src/nodes/store_astra/IGlobal.pynodes/src/nodes/store_astra/IInstance.pynodes/src/nodes/store_astra/README.mdnodes/src/nodes/store_astra/__init__.pynodes/src/nodes/store_astra/astra_db.pynodes/src/nodes/store_astra/requirements.txtnodes/src/nodes/store_astra/services.jsonnodes/src/nodes/store_atlas/IEndpoint.pynodes/src/nodes/store_atlas/IGlobal.pynodes/src/nodes/store_atlas/IInstance.pynodes/src/nodes/store_atlas/README.mdnodes/src/nodes/store_atlas/__init__.pynodes/src/nodes/store_atlas/atlas.pynodes/src/nodes/store_atlas/requirements.txtnodes/src/nodes/store_atlas/services.jsonnodes/src/nodes/store_chroma/IEndpoint.pynodes/src/nodes/store_chroma/IGlobal.pynodes/src/nodes/store_chroma/IInstance.pynodes/src/nodes/store_chroma/README.mdnodes/src/nodes/store_chroma/__init__.pynodes/src/nodes/store_chroma/chroma.pynodes/src/nodes/store_chroma/requirements.txtnodes/src/nodes/store_chroma/services.jsonnodes/src/nodes/store_elasticsearch/IEndpoint.pynodes/src/nodes/store_elasticsearch/IGlobal.pynodes/src/nodes/store_elasticsearch/IInstance.pynodes/src/nodes/store_elasticsearch/README.mdnodes/src/nodes/store_elasticsearch/__init__.pynodes/src/nodes/store_elasticsearch/constants.pynodes/src/nodes/store_elasticsearch/elasticsearch/README.mdnodes/src/nodes/store_elasticsearch/elasticsearch_store.pynodes/src/nodes/store_elasticsearch/opensearch/README.mdnodes/src/nodes/store_elasticsearch/opensearch_client.pynodes/src/nodes/store_elasticsearch/requirements.txtnodes/src/nodes/store_elasticsearch/services.elasticsearch.jsonnodes/src/nodes/store_elasticsearch/services.opensearch.jsonnodes/src/nodes/store_milvus/IEndpoint.pynodes/src/nodes/store_milvus/IGlobal.pynodes/src/nodes/store_milvus/IInstance.pynodes/src/nodes/store_milvus/README.mdnodes/src/nodes/store_milvus/__init__.pynodes/src/nodes/store_milvus/milvus.pynodes/src/nodes/store_milvus/requirements.txtnodes/src/nodes/store_milvus/services.jsonnodes/src/nodes/store_pinecone/IEndpoint.pynodes/src/nodes/store_pinecone/IGlobal.pynodes/src/nodes/store_pinecone/IInstance.pynodes/src/nodes/store_pinecone/README.mdnodes/src/nodes/store_pinecone/__init__.pynodes/src/nodes/store_pinecone/pinecone.pynodes/src/nodes/store_pinecone/requirements.txtnodes/src/nodes/store_pinecone/services.jsonnodes/src/nodes/store_postgres/IEndpoint.pynodes/src/nodes/store_postgres/IGlobal.pynodes/src/nodes/store_postgres/IInstance.pynodes/src/nodes/store_postgres/README.mdnodes/src/nodes/store_postgres/__init__.pynodes/src/nodes/store_postgres/postgres.pynodes/src/nodes/store_postgres/requirements.txtnodes/src/nodes/store_postgres/services.jsonnodes/src/nodes/store_qdrant/IEndpoint.pynodes/src/nodes/store_qdrant/IGlobal.pynodes/src/nodes/store_qdrant/IInstance.pynodes/src/nodes/store_qdrant/README.mdnodes/src/nodes/store_qdrant/__init__.pynodes/src/nodes/store_qdrant/qdrant.pynodes/src/nodes/store_qdrant/requirements.txtnodes/src/nodes/store_qdrant/services.jsonnodes/src/nodes/store_weaviate/IEndpoint.pynodes/src/nodes/store_weaviate/IGlobal.pynodes/src/nodes/store_weaviate/IInstance.pynodes/src/nodes/store_weaviate/README.mdnodes/src/nodes/store_weaviate/__init__.pynodes/src/nodes/store_weaviate/requirements.txtnodes/src/nodes/store_weaviate/services.jsonnodes/src/nodes/store_weaviate/weaviate.pynodes/test/astra_db/test_convert_filter.pynodes/test/atlas/test_isdeleted_filter.pynodes/test/chroma/test_coerce_port.pynodes/test/chroma/test_convert_filter_explicit_none.pynodes/test/conftest.pynodes/test/milvus/test_addchunks_return.pynodes/test/milvus/test_filter_injection.pynodes/test/milvus/test_search_semantic.pynodes/test/pinecone/test_record_updates.pynodes/test/qdrant/test_mark_deleted.pynodes/test/qdrant/test_search_threshold.pynodes/test/test_db_arango.pynodes/test/test_db_hydradb.pynodes/test/weaviate/test_record_updates.pypackages/docs/redirects.ts
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
🛑 Comments failed to post (51)
nodes/src/nodes/graph_arango/IGlobal.py (1)
184-204: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
affected_rowsis silently zeroed out whenever a write query also returns rows.
_run_query_rawreturns'affected_rows': 0 if rows else affected— so anINSERT ... RETURN NEW(or any write with aRETURNclause) always reportsaffected_rows: 0, even though documents were actually written. This value flows unmodified into theanswerslane JSON payload inIInstance.writeQuestions's EXECUTE handler, so an SDK caller relying specifically onaffected_rows(rather than countingrows) gets a misleading zero for a successful write.len(rows)would be the more accurate affected count.This requires two independent preconditions to surface (
allow_execute=true, non-default, and a write query withRETURN), consistent with a previously flagged pattern in a sibling graph node, so Major rather than Critical severity applies here too.🐛 Proposed fix
- affected = _affected_rows(cursor) - return {'rows': rows, 'affected_rows': 0 if rows else affected} + affected = len(rows) if rows else _affected_rows(cursor) + return {'rows': rows, 'affected_rows': affected}Based on learnings: "downgrade an issue's severity from Critical to Major if the bug can only be triggered when multiple independent preconditions align—such as a non-default config flag (e.g., allow_execute) being enabled, the executed query having a specific shape (e.g., write + RETURN), and downstream/caller logic relying on a specific returned field for retry logic (e.g., affected_rows)."
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def _run_query_raw(self, aql: str, *, max_runtime: float = QUERY_MAX_RUNTIME) -> Dict[str, Any]: """Execute a raw AQL statement without the ``_is_aql_safe`` gate. Used by the EXECUTE path where the caller has accepted the risk of running write AQL directly. Returns ``{'rows': [...], 'affected_rows': N}`` to mirror the SQL ``_executeRawQuery`` shape — ``affected_rows`` is derived from the cursor statistics when no rows are returned (e.g. INSERT without RETURN). Raises: arango.exceptions.ArangoError: Caught at the IInstance handler per precedent. """ max_rows = self.max_execute_rows cursor = self.db.aql.execute(aql, count=False, max_runtime=max_runtime, memory_limit=self.QUERY_MEMORY_LIMIT) rows: List[Dict] = [] for doc in cursor: # Check before appending (mirrors _run_query) so we never hold an extra row. if len(rows) >= max_rows: raise ValueError(f'EXECUTE query exceeded max_execute_rows={max_rows}') rows.append(doc) affected = len(rows) if rows else _affected_rows(cursor) return {'rows': rows, 'affected_rows': affected}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/graph_arango/IGlobal.py` around lines 184 - 204, Update _run_query_raw so affected_rows reflects writes even when the query returns rows: use len(rows) as the affected count when rows are present, while retaining the cursor-derived affected count when no rows are returned. Keep the existing row collection and max_execute_rows enforcement unchanged.Source: Learnings
nodes/src/nodes/graph_arango/IInstance.py (1)
175-224: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
get_aql'sinput_schemadoesn't document thelimitparameter it actually accepts.
get_aqlreadsargs.get('limit')(line 210) andget_datarelies on this by passinglimitthrough internally (line 104), butget_aql's owninput_schema(lines 176-185) only declaresquestion. An agent callingget_aqldirectly via the tool-invoke path has no documented way to knowlimitis supported.♻️ Proposed fix
input_schema={ 'type': 'object', 'required': ['question'], 'properties': { 'question': { 'type': 'string', 'description': 'Natural-language question to convert into an AQL query', }, + 'limit': { + 'type': 'integer', + 'description': f'Row limit to embed in the generated AQL LIMIT clause (default 250, max {DEFAULT_MAX_EXECUTE_ROWS}).', + }, }, },📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.`@tool_function`( input_schema={ 'type': 'object', 'required': ['question'], 'properties': { 'question': { 'type': 'string', 'description': 'Natural-language question to convert into an AQL query', }, 'limit': { 'type': 'integer', 'description': f'Row limit to embed in the generated AQL LIMIT clause (default 250, max {DEFAULT_MAX_EXECUTE_ROWS}).', }, }, }, output_schema={ 'type': 'object', 'properties': { 'aql': {'type': 'string', 'description': 'The generated read-only AQL query.'}, 'valid': {'type': 'boolean', 'description': 'Whether a valid, safe AQL query was generated.'}, 'error': {'type': 'string', 'description': 'Error message if the generated AQL was unsafe.'}, 'answer': { 'type': 'string', 'description': 'LLM text response when the question is not a database query.', }, }, }, description=( 'Accepts a natural-language description and returns the equivalent read-only AQL query without executing it. Only use when the user explicitly asks to see the AQL — for actual data retrieval, use get_data instead.' ), ) def get_aql(self, args): """Translate natural language to AQL without executing.""" if not isinstance(args, dict): raise ValueError('Tool input must be a JSON object') question = args.get('question') if not question or not isinstance(question, str) or not question.strip(): raise ValueError('"question" is required and must be a non-empty string') limit = _clamp_limit(args.get('limit')) result = self._buildAqlQuery(question.strip(), limit=limit) is_valid = _parse_is_valid(result.get('isValid', False)) aql = result.get('query', '') # is_valid means the query passed the EXPLAIN-plan read-only gate in # _buildAqlQuery, so it is already known safe — no keyword re-check here. if is_valid and aql: return {'aql': aql, 'valid': True} # Not valid: an EXPLAIN-validation failure carries an 'error'; a genuine # off-topic question does not (its 'query' is the LLM's plain-text answer). if result.get('error'): return {'error': result['error'], 'aql': aql, 'valid': False} return {'answer': aql, 'valid': False}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/graph_arango/IInstance.py` around lines 175 - 224, Update the get_aql tool’s input_schema to document the supported limit parameter alongside question, including its expected numeric type and purpose, while preserving the existing _clamp_limit(args.get('limit')) behavior.nodes/src/nodes/graph_arango/README.md (1)
1-1: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Stale
db_arangoreferences left over from the rename.The H1 title (line 1: "# db_arango") and the "View source" link (line 104, pointing to
.../nodes/src/nodes/db_arango) still reference the old node name/path, even though this file now lives atnodes/src/nodes/graph_arango/README.mdand the module path was updated elsewhere in this PR (services.json'spath→nodes.graph_arango).📝 Proposed fix
-# db_arango +# graph_arango-[<svg ...>View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/db_arango) +[<svg ...>View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/graph_arango)Also applies to: 104-104
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/graph_arango/README.md` at line 1, Update the README H1 and its “View source” link to use the renamed graph_arango node and path, replacing the stale db_arango references while preserving the existing link structure.nodes/src/nodes/store_astra/astra_db.py (6)
62-90: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
api_endpointis neither normalized nor validated as required.The comment at line 70 claims stripping of spaces,
http(s)://, and trailing slashes, but line 71 just doesconfig.get('api_endpoint')with no processing at all — the comment is stale/misleading. Separately, unlikecollection_name(regex-validated) andsimilarity(whitelist-validated), there's no check thatapi_endpointis even present; a missing value silently becomesNoneand is passed straight intoself.client.get_database(None, ...), producing an opaque astrapy error instead of a clear configuration error.🐛 Proposed fix
- # Remove leading and trailing spaces, leading http/https and :// and trailing slashes - self.api_endpoint = config.get('api_endpoint') + self.api_endpoint = config.get('api_endpoint') + if not self.api_endpoint: + raise Exception('Missing required config: api_endpoint') + self.api_endpoint = re.sub(r'^https?://', '', self.api_endpoint.strip()).rstrip('/')📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.config = Config.getNodeConfig(provider, connConfig) # Save our parameters self.collection_name = config.get('collection', '') if not re.match(r'^[A-Za-z0-9][A-Za-z0-9_]*$', self.collection_name): raise Exception(f'Invalid collection name: {self.collection_name}') self.api_endpoint = config.get('api_endpoint') if not self.api_endpoint: raise Exception('Missing required config: api_endpoint') self.api_endpoint = re.sub(r'^https?://', '', self.api_endpoint.strip()).rstrip('/') # check if the similarity matches astra configuration options similarity = config.get('similarity', 'cosine') if similarity in ['cosine', 'euclidean', 'dot_product']: self.similarity = similarity else: raise Exception( 'The similarity metric you provided in the config.json does not match required astra configurations' ) # Strip API key also self.application_token = config.get('application_token', None) if self.application_token is not None: self.application_token = self.application_token.strip() # Init the store self.client = DataAPIClient() self.database = self.client.get_database(self.api_endpoint, token=self.application_token) return🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_astra/astra_db.py` around lines 62 - 90, Update api_endpoint initialization in the store setup to require a configured value, normalize it by trimming whitespace, removing the http/https scheme and surrounding trailing slashes as described by the existing comment, and validate the resulting endpoint before passing it to get_database. Raise a clear configuration error for missing or invalid endpoints, while preserving the existing collection_name, similarity, and application_token handling.
147-200: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Unescaped user/path value injected into
$regexfilter.
docFilter.parentis inserted directly into{'$regex': docFilter.parent}with no escaping. Parent paths commonly contain regex metacharacters (.,(,),+, etc.) — e.g..matches any character — so a path like/docs/report.pdfcan incidentally match/docs/reportXpdf, silently returning wrong documents.🐛 Proposed fix
if docFilter.parent is not None: - filter_dict['meta.parent'] = {'$regex': docFilter.parent} + filter_dict['meta.parent'] = {'$regex': re.escape(docFilter.parent)}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_astra/astra_db.py` around lines 147 - 200, Escape the user-supplied parent path before assigning it to the $regex filter in _convertFilter. Use the standard regex-escaping utility so metacharacters in docFilter.parent are matched literally, while preserving the existing parent filter behavior and all other filter clauses.
276-299: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Missing embedding-null and non-zero-offset guards, unlike sibling stores.
Every other store's
searchSemanticin this codebase (Elasticsearch, Milvus — see supplied graph context) explicitly checksif query.embedding is None: raise Exception(...)and rejects non-zerodocFilter.offsetwith a clear error, since semantic/vector search paging by offset generally isn't supported. Astra's implementation does neither: a missing embedding will fail deep inside astrapy with an unclear error, and a non-zero offset is silently ignored (callers expecting pagination will always get the same first page).Based on learnings, this mirrors the established pattern used elsewhere (e.g., `nodes/src/nodes/store_elasticsearch/elasticsearch_store.py` `searchSemantic`).🐛 Proposed fix
filter_dict = self._convertFilter(docFilter) + + # Check embedding + if query.embedding is None: + raise Exception('To use semantic search, you must bind to an embedding module') + + # We cannot support non-zero offsets + if docFilter.offset: + raise BaseException('Non-zero offset is not supported in semantic searching')📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def searchSemantic(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]: """ Semantic search. """ if not self._doesCollectionExist(): return [] filter_dict = self._convertFilter(docFilter) # Check embedding if query.embedding is None: raise Exception('To use semantic search, you must bind to an embedding module') # We cannot support non-zero offsets if docFilter.offset: raise BaseException('Non-zero offset is not supported in semantic searching') # Combine both lexical and vector search cursor = self.database.get_collection(self.collection_name).find( filter_dict, sort={'$vector': query.embedding}, # Automatic embedding generation + search limit=docFilter.limit, projection={'$lexical': 0}, include_similarity=True, include_sort_vector=True, ) # Convert results to your Doc format docs = self._convertToDocs(cursor) return docs🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 285-292: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: self.database.get_collection(self.collection_name).find(
filter_dict,
sort={'$vector': query.embedding}, # Automatic embedding generation + search
limit=docFilter.limit,
projection={'$lexical': 0},
include_similarity=True,
include_sort_vector=True,
)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').(xpath-injection-python)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_astra/astra_db.py` around lines 276 - 299, Update searchSemantic to raise a clear exception when query.embedding is None and reject any non-zero docFilter.offset before querying Astra. Preserve the existing collection check and semantic search flow for valid embeddings with offset zero.
318-357: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
getPathsbypasses the default soft-delete exclusion.Unlike every other read path in this file,
getPathsbuilds its ownmatch_filterdirectly instead of going through_convertFilter, so it never addsmeta.isDeleted: False. The README explicitly documents that soft-deleted chunks are "excluded from every query unless the filter explicitly requests them" —getPathsviolates that, returning parent/objectId pairs for soft-deleted objects too.🐛 Proposed fix
- match_filter = {'meta.chunkId': 0} + match_filter = {'meta.chunkId': 0, 'meta.isDeleted': False}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def getPaths(self, parent: str | None = None, offset: int = 0, limit: int = 1000) -> Dict[str, str]: """ More efficient implementation using aggregation to get distinct parent paths. """ if not self._doesCollectionExist(): return {} # Build match stage filter match_filter = {'meta.chunkId': 0, 'meta.isDeleted': False} if parent is not None: match_filter['meta.parent'] = parent # Use aggregation pipeline pipeline = [ {'$match': match_filter}, { '$group': { '_id': '$meta.parent', 'objectId': {'$first': '$meta.objectId'}, } }, {'$skip': offset}, {'$limit': limit}, ] # Execute aggregation collection = self.database.get_collection(self.collection_name) results = collection.aggregate(pipeline) # Build paths dictionary paths: Dict[str, str] = {} for result in results: parent_path = result['_id'] object_id = result['objectId'] if parent_path and object_id: paths[parent_path] = object_id return paths🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_astra/astra_db.py` around lines 318 - 357, Update getPaths to exclude soft-deleted records by adding the standard meta.isDeleted: False condition to its aggregation match filter, while preserving the existing parent filter, pagination, and path-building behavior.
358-435: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win
Critical: the near-zero-vector filter drops the boosted control/schema document for realistic embedding sizes.
is_valid_vectorrequiresmagnitude > 1e-6. The control-document workaround boosts a zero embedding to[1e-8] * len(embedding), givingmagnitude = 1e-8 * sqrt(len(embedding)). For this to exceed1e-6,len(embedding)must exceed 10,000 — but real embedding dimensions (384–3072) all fall far below that (e.g. at 1536 dims, magnitude ≈ 3.9e-7). So for essentially every real embedding model,valid_docsexcludes the'schema'control document, and it never gets inserted — silently defeating the special-case handling and, perdoesCollectionExist's docstring, breaking the "control document" detection that base-class collection-fix logic depends on.🐛 Proposed fix
- def is_valid_vector(vector): - magnitude = sum(x * x for x in vector) ** 0.5 - return magnitude > 1e-6 - - valid_docs = [doc for doc in documents if is_valid_vector(doc.get('$vector', [1]))] + def is_valid_vector(doc): + if doc.get('meta', {}).get('objectId') == 'schema': + return True + vector = doc.get('$vector', [1]) + magnitude = sum(x * x for x in vector) ** 0.5 + return magnitude > 1e-6 + + valid_docs = [doc for doc in documents if is_valid_vector(doc)]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def addChunks(self, chunks: List[Doc], checkCollection: bool = True) -> None: """ Add document chunks to the document store. """ if not chunks: return if checkCollection and not self.createCollection(chunks): return collection = self.database.get_collection(self.collection_name) # Clear the documents list documents: List[Dict] = [] # Clear the object id list for deletion objectIds: Dict = {} def flush(): nonlocal documents nonlocal objectIds # First, delete existing documents if we have objectIds to delete if len(objectIds): # Delete all documents with these object IDs for objectId in objectIds.keys(): collection.delete_many({'meta.objectId': objectId}) def is_valid_vector(doc): if doc.get('meta', {}).get('objectId') == 'schema': return True vector = doc.get('$vector', [1]) magnitude = sum(x * x for x in vector) ** 0.5 return magnitude > 1e-6 valid_docs = [doc for doc in documents if is_valid_vector(doc)] # Insert the new documents if len(valid_docs): collection.insert_many(valid_docs) # Clear them objectIds = {} documents = [] # For each document, check if we need to delete existing chunks for chunk in chunks: # If we are writing chunk 0, then delete the object if not chunk.metadata.chunkId: # Save this object id to be deleted objectIds[chunk.metadata.objectId] = True # For each document, prepare the data for insertion for chunk in chunks: # Get the embedding embedding = chunk.embedding # If we do not have an embedding if embedding is None: raise Exception('No embedding in document') # Handle control document with zero vectors if chunk.metadata.objectId == 'schema' and all(x == 0 for x in embedding): embedding = [1e-8] * len(embedding) # Create the document structure for AstraDB document = { '_id': str(uuid4()), '$vector': DataAPIVector(embedding), 'content': chunk.page_content, 'meta': chunk.metadata.__dict__, # Convert metadata to dict } documents.append(document) if len(documents) >= 500: flush() # Flush any remaining documents flush()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_astra/astra_db.py` around lines 358 - 435, Update the near-zero-vector validation inside addChunks.flush so the schema control document created by the zero-embedding workaround is retained for insertion. Ensure the validity check or schema-specific handling accepts that boosted control vector while continuing to filter genuinely invalid vectors for other documents, preserving doesCollectionExist control-document detection.
490-529: 🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
renderloads the entire object into memory in one shot, contradicting its own comment and diverging from sibling stores.The comment above
render(lines 490-493) says chunks are gathered "one group at a time (renderChunkSize)" and the callback is invoked per group — matching how Atlas, Postgres, and Milvus actually implementrender(looping withskip/limitbyrenderChunkSize, callingcallbackper batch). Astra's implementation instead fetches all chunks for the object unconditionally, sorts them all in memory, joins them into one string, and callscallbackexactly once. There is norenderChunkSizeattribute on thisStoreat all. For objects with tens of thousands of chunks (explicitly called out as the target case), this risks large memory spikes/OOM and is inconsistent with the rest of the store fleet's render contract.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_astra/astra_db.py` around lines 490 - 529, Update Store.render to process object chunks in batches using the store’s renderChunkSize configuration, rather than collecting, sorting, joining, and callbacking the entire object at once. Preserve chunk ordering by fetching each batch in chunkId order, invoke callback once per batch, and add or reuse the appropriate renderChunkSize attribute consistent with sibling stores.nodes/src/nodes/store_astra/README.md (1)
84-99: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Generated "Source" link still points to the pre-rename
astra_dbpath.The generated block links to
nodes/src/nodes/astra_db, but this node now lives atnodes/src/nodes/store_astra. As per coding guidelines, content in this block must not be hand-edited since it is "regenerated bynodes:docs-generate" — please re-run the docs generator against the renamed path before merge so this link resolves correctly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_astra/README.md` around lines 84 - 99, Regenerate the documentation for the renamed store_astra node using the nodes:docs-generate workflow so the generated Source link points to nodes/src/nodes/store_astra instead of the obsolete astra_db path. Do not manually edit the generated block.Source: Coding guidelines
nodes/src/nodes/store_atlas/atlas.py (5)
236-249: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Redundant, inconsistent collection-existence re-check in
searchSemantic.Line 238 already returns
[]if the collection is missing. Line 242 re-checks withself.doesCollectionExist(query.embedding_model)and raises instead — but_doesCollectionExist()(L131-135) takes no parameters and never inspects an embedding model, so the comment "Validate embedding model matches collection" doesn't reflect what's actually being checked. This looks like leftover/dead logic that produces confusing, inconsistent error behavior (silent[]vs raised exception) for what is effectively the same failure condition.🐛 Suggested cleanup
- if not self.doesCollectionExist(): - return [] - - # Validate embedding model matches collection - if not self.doesCollectionExist(query.embedding_model): - raise Exception('Collection does not exist or model mismatch') - - if query.embedding is None: + if not self.doesCollectionExist(): + return [] + + if query.embedding is None: raise Exception('Embedding required for semantic search')📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.def searchSemantic(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]: """Perform semantic search using Atlas Vector Search.""" if not self.doesCollectionExist(): return [] if query.embedding is None: raise Exception('Embedding required for semantic search') if docFilter.offset: raise Exception('Non-zero offset not supported in semantic search')🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_atlas/atlas.py` around lines 236 - 249, Remove the redundant model-parameter collection check and its misleading validation comment from searchSemantic. Keep the initial doesCollectionExist() guard returning [] for missing collections, while preserving the separate embedding and offset validations.
270-276: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
UnboundLocalErrorwhen the aggregation raisesOperationFailure.
resultsis only assigned inside thetryblock. Ifself.collection.aggregate(pipeline)raisesOperationFailure, theexceptbranch only logs the error and falls through —resultsis never bound, and the subsequentreturn self._convertToDocs(results, ...)crashes withUnboundLocalErrorinstead of failing gracefully.🐛 Proposed fix
+ results = [] try: results = list(self.collection.aggregate(pipeline)) logger.info(f'Results: {results}') except OperationFailure as e: logger.error(f'OperationFailure: {e}') + return [] return self._convertToDocs(results, include_score=True)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.results = [] try: results = list(self.collection.aggregate(pipeline)) logger.info(f'Results: {results}') except OperationFailure as e: logger.error(f'OperationFailure: {e}') return [] return self._convertToDocs(results, include_score=True)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_atlas/atlas.py` around lines 270 - 276, Initialize results to a safe empty collection before the try block, or return an appropriate empty-document result directly from the OperationFailure handler, so _convertToDocs receives a defined value when aggregate fails. Preserve the existing successful aggregation and logging behavior.
294-316: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== file outline ==" ast-grep outline nodes/src/nodes/store_atlas/atlas.py --view expanded || true echo "== relevant sections ==" cat -n nodes/src/nodes/store_atlas/atlas.py | sed -n '260,330p' printf '\n---\n' cat -n nodes/src/nodes/store_atlas/atlas.py | sed -n '420,495p' echo "== searches for parent/docFilter source ==" rg -n "def getPaths|parent|docFilter|_convertFilter|get_paths|docsFilter|storeCollection|getCollection|filter_conditions" nodes/src/nodes/store_atlas/atlas.py echo "== imports ==" cat -n nodes/src/nodes/store_atlas/atlas.py | sed -n '1,80p'Repository: rocketride-org/rocketride-server
Length of output: 15588
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== locate schema/store definitions ==" git ls-files | rg 'ai/common/schema\.py$|ai/common/store\.py$|schema\.py$|store\.py$' || true echo "== DocFilter occurrences ==" rg -n "class DocFilter|DocFilter|class DocumentStoreBase|def getPaths|getDocs|searchKeyword|searchSemantic|get\(" . echo "== schema section around DocFilter ==" for f in $(git ls-files | rg 'ai/common/schema\.py$|ai/common/store\.py$'); do echo "--- $f ---" cat -n "$f" | sed -n '1,220p' doneRepository: rocketride-org/rocketride-server
Length of output: 50389
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== candidate schema/store files ==" git ls-files | rg '(^|/)schema\.py$|(^|/)store\.py$' | sed -n '1,120p' echo "== focused DocFilter definitions ==" rg -n "class DocFilter|DocFilter =|parent:.*str|parent.*str|metadata" ai/common/schema.py ai/common/store.py 2>/dev/null | sed -n '1,160p' || true echo "== focused schema/store snippets ==" for f in ai/common/schema.py.ai/common/store.py; do if [ -f "$f" ]; then echo "--- $f ---" cat -n "$f" | sed -n '1,260p' fi doneRepository: rocketride-org/rocketride-server
Length of output: 462
🏁 Script executed:
#!/bin/bash set -euo pipefail for f in packages/ai/src/ai/common/store.py packages/ai/src/ai/common/schema.py; do if [ -f "$f" ]; then echo "--- $f ---" wc -l "$f" cat -n "$f" | sed -n '1,260p' else echo "MISSING: $f" fi done echo "== imports and API paths mentioning getPaths / DocFilter ==" rg -n "from ai\.common\.schema import|import .*schema|DocFilter|getPaths\(|searchKeyword|searchSemantic|\.get\(" packages nodes | sed -n '1,240p'Repository: rocketride-org/rocketride-server
Length of output: 38951
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== shell-api DocFilter/type definitions ==" if [ -f packages/shell-api/versions/v0.d.ts ]; then cat -n packages/shell-api/versions/v0.d.ts | sed -n '70,100p;150,185p' fi echo "== TypeScript/TSX parent-field usages ==" rg -n "DocFilter|filter.*parent|parent.*filter|getPaths|\bparent\b" packages shared-ui 2>/dev/null | sed -n '1,260p' || true echo "== Python search/store wrappers around getPaths/search calls ==" rg -n "getPaths\(|searchKeyword\(|searchSemantic\(|DocFilter\(" nodes/src packages/ai/src packages/client-python/src 2>/dev/null | sed -n '1,240p' || trueRepository: rocketride-org/rocketride-server
Length of output: 40644
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== ai/common/schema.py files ==" git ls-files | rg 'packages/.*/src/ai/common/schema\.py$|ai/common/schema\.py$' || true echo "== ai/common/schema.py doc filter excerpt ==" for f in $(git ls-files | rg 'packages/.*/src/ai/common/schema\.py$|ai/common/schema\.py$'); do echo "--- $f ---" rg -n "class DocFilter|parent|Field\(|pydantic|BaseModel" "$f" | sed -n '1,180p' if rg -n "class DocFilter" "$f" >/dev/null; then line=$(rg -n "^class DocFilter" "$f" | head -1 | cut -d: -f1 || true) if [ -n "${line:-}" ]; then start=$(( line >= 1 ? line-20 : 1 )) end=$(( line + 180 )) cat -n "$f" | sed -n "${start},${end}p" fi fi done echo "== schema converter around parent doc filter ==" for f in $(git ls-files | rg 'packages/.*/src/ai/common/schema\.py$|ai/common/schema\.py$'); do rg -n "docFilter|DocFilter|parent|convert|toDict|fromDict|search" "$f" | sed -n '1,240p' echo "--- $f converter sections ---" rg -n "def .*doc.*filter|def .*filter|doc_filters|filter_schema|Question|parent" "$f" | while read -r match; do line=$(echo "$match" | cut -d: -f1) echo "at line $line" start=$(( line >= 40 ? line-40 : 1 )) end=$(( line + 120 )) cat -n "$f" | sed -n "${start},${end}p" echo "------------" done || true done echo "== shell api doc filter implementation if present ==" fd -a 'DocFilter\.(ts|js)$|DocFilter\.d\.ts$|schema' packages/shell-api packages/shared-ui | sed -n '1,80p' || trueRepository: rocketride-org/rocketride-server
Length of output: 348
Escape
parentvalues before MongoDB regex filtering.
getPaths()and_convertFilter()passparent/docFilter.parentdirectly into$regexfilters. If those values can originate from external/tool input, escaping them (e.g.re.escape(...)) prevents regex-injection-style filter bypass and costly pattern backtracking.🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 302-302: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: self.collection.find(filter_conditions)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').(xpath-injection-python)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_atlas/atlas.py` around lines 294 - 316, Escape externally supplied parent filter values before constructing MongoDB `$regex` conditions in both `getPaths()` and `_convertFilter()`. Apply `re.escape(...)` to `parent` and `docFilter.parent` while preserving the existing case-insensitive matching behavior and filter structure.
318-353: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Existing documents are deleted before every chunk's embedding is validated.
objectIdsare computed and deleted (L327-334) before the loop that raisesException('No embedding in document')(L341-342) for any chunk missing an embedding. If a later chunk in the batch lacks an embedding, the pre-existing documents for thatobjectIdhave already been removed and the exception aborts before the replacement documents are inserted — leaving the object with no data in the store.Validate that every chunk has an embedding before deleting the pre-existing documents for the same object IDs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_atlas/atlas.py` around lines 318 - 353, Update addChunks to validate that every chunk has a non-None embedding before computing and deleting existing object IDs. Move the embedding validation ahead of the objectIds deletion block, while preserving the existing exception behavior and document preparation flow.
351-352: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
sys.getsizeof(doc)does not measure real document size.
sys.getsizeofonly returns the shallow size of thedictcontainer, not the size of the nestedembeddinglist orcontentstring it holds. Thepayload_limit-based flush branch (L356) is effectively driven only by the 500-document count check, sincesum_sizewill almost never reflect actual payload bytes — risking oversized batches when embeddings/content are large.Consider estimating size via
len(content.encode()) + len(embedding) * 8(or BSON-encoding the doc) instead ofsys.getsizeof.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_atlas/atlas.py` around lines 351 - 352, Replace the shallow sys.getsizeof(doc) calculation in the document batching loop with an estimate of the serialized payload that includes content bytes and embedding element storage, or BSON-encode the document for an exact measure. Keep accumulating the estimate in sum_size so the existing payload_limit flush condition reflects actual document size.nodes/src/nodes/store_atlas/README.md (1)
74-77: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Generated "View source" link still points to the old
atlaspath.The link targets
.../tree/develop/nodes/src/nodes/atlas, but the node now lives undernodes/src/nodes/store_atlas. Since this is inside theROCKETRIDE:GENERATED:PARAMSblock, re-runnodes:docs-generateagainst the renamed directory rather than hand-editing it.As per coding guidelines, "do not hand-edit content between
<!-- ROCKETRIDE:GENERATED:PARAMS START -->and<!-- ROCKETRIDE:GENERATED:PARAMS END -->as it is regenerated bynodes:docs-generate" — please regenerate rather than patch this line directly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_atlas/README.md` around lines 74 - 77, Regenerate the documentation for the renamed store_atlas node using nodes:docs-generate so the generated “View source” link targets nodes/src/nodes/store_atlas. Do not manually edit content within the ROCKETRIDE:GENERATED:PARAMS block.Source: Coding guidelines
nodes/src/nodes/store_chroma/chroma.py (4)
245-253: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
chromadb collection query cosine distance range hnsw:space💡 Result:
In ChromaDB, when you configure a collection with
hnsw:space: "cosine", the system calculates and returns "cosine distance," not standard cosine similarity [1][2]. Cosine distance is derived from cosine similarity using the formula: Distance = 1.0 - Cosine Similarity [3][4][2]. Because of this definition, the range of values returned by ChromaDB for cosine distance is: - 0.0: Represents identical vectors (maximum similarity) [4][2][5]. - 1.0: Represents orthogonal vectors (no similarity) [4][5]. - 2.0: Represents opposite vectors (maximum dissimilarity) [2]. Key implementation details include: - Configuration: You specify the distance metric at collection creation time usingconfiguration={"hnsw": {"space": "cosine"}}or the legacy metadata approachmetadata={"hnsw:space": "cosine"}[1][6][2]. This setting cannot be changed after the collection is created [6]. - Interaction: When querying a collection configured withcosine, thedistancesfield in the query result will contain these cosine distance values [1][7][8]. - Best Practices: Since ChromaDB returns distance (where lower is closer/better), you should interpret the results accordingly [9][2]. If your application requires a traditional similarity score (ranging from -1 to 1), you can convert the ChromaDB distance back to similarity by calculating (1.0 - distance) [2][5].Citations:
- 1: https://cookbook.chromadb.dev/core/collections/
- 2: https://topictrick.com/blog/chromadb-tutorial-beginner-guide
- 3: https://docs.trychroma.com/docs/collections/configure
- 4: https://stackoverflow.com/questions/77794024/searching-existing-chromadb-database-using-cosine-similarity
- 5: https://cookbook.chromadb.dev/faq/
- 6: https://cookbook.chromadb.dev/core/configuration/
- 7: https://chroma-core-chroma.mintlify.app/api/python/collection
- 8: https://github.com/chroma-core/chroma/blob/main/chromadb/api/models/Collection.py
- 9: https://cookbook.chromadb.dev/core/advanced/queries/
🏁 Script executed:
#!/bin/bash set -euo pipefail file="nodes/src/nodes/store_chroma/chroma.py" if [ -f "$file" ]; then echo "== file size ==" wc -l "$file" echo echo "== relevant excerpts ==" sed -n '1,120p' "$file" | cat -n echo sed -n '120,290p' "$file" | cat -n -v echo echo "== cosine/distances occurrences ==" rg -n "similarity|distances|cosine|embedding|query" "$file" -C 3 else echo "missing $file" git ls-files | rg 'chroma\.py$' fiRepository: rocketride-org/rocketride-server
Length of output: 20134
Normalize Chroma cosine scores in the same direction as the distance.
Chroma returns
distancesfor cosine as1 - cosine_similarityin[0, 2], so better matches have lower distances. The current mapping(distance + 1) / 2makes a perfect match score0.5and an opposite vector score1.5; apply the score thresholdself.threshold_searchwith this score ordering (or convert to1 - distance/2before thresholding).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_chroma/chroma.py` around lines 245 - 253, Update the cosine branch in the score calculation around self.similarity to convert Chroma’s distance into a similarity score using 1 - distance / 2, so lower distances produce higher scores. Apply self.threshold_search to this normalized score instead of the hardcoded 0.20 threshold, while preserving the existing non-cosine scoring path.
374-378: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== locate file ==" git ls-files | grep -F 'nodes/src/nodes/store_chroma/chroma.py' || true echo "== outline relevant file ==" ast-grep outline nodes/src/nodes/store_chroma/chroma.py --match getPaths --view expanded || true ast-grep outline nodes/src/nodes/store_chroma/chroma.py --match _convertFilter --view expanded || true echo "== relevant sections ==" sed -n '330,410p' nodes/src/nodes/store_chroma/chroma.py echo "---" sed -n '1,180p' nodes/src/nodes/store_chroma/chroma.py echo "== search related filter usage ==" rg -n "chunkId|filter_dict|getPaths|_convertFilter|parent" nodes/src/nodes/store_chroma/chroma.pyRepository: rocketride-org/rocketride-server
Length of output: 11643
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== chromadb dependency versions ==" fd -a 'requirements.txt|pyproject.toml|poetry.lock|Pipfile.lock' . | sed 's#^\./##' | while read -r f; do if grep -qiE 'chromad( |b|b\.)|chroma' "$f"; then echo "--- $f" grep -n -iE 'chromad( |b|b\.)|chroma' "$f" || true fi done echo "== relevant filter helper ==" sed -n '180,230p' nodes/src/nodes/store_chroma/chroma.py echo "== relevant add chunk storage ==" sed -n '420,590p' nodes/src/nodes/store_chroma/chroma.py echo "== tests/usages of getPaths and chunkId ==" rg -n "getPaths|chunkId|chunk\":|'chunk'|\"chunk\"" . | head -200Repository: rocketride-org/rocketride-server
Length of output: 35392
🌐 Web query:
ChromaDB 0.5 Python client get where metadata multiple keys top-level where two keys $and💡 Result:
To filter by multiple metadata keys using the $and operator in the ChromaDB Python client, you must provide a dictionary to the where argument with the $and key associated with a list of condition objects [1][2]. Each condition object in the list should define a single metadata field filter [2]. Example structure for filtering by two keys: collection.get( where={ "$and": [ {"key1": "value1"}, {"key2": {"$gte": 10}} ] }) Key points for implementation: 1. Use a List for $and: The $and operator requires a list of dictionaries [1][2]. Each dictionary in that list must contain exactly one field condition [2]. 2. Comparison Operators: You can mix simple equality checks (e.g., {"key1": "value1"}) with more complex operators like $gte, $lte, $contains, etc. (e.g., {"key2": {"$gte": 10}}) [1][3][4]. 3. Alternative (K-Expressions): ChromaDB also supports a cleaner syntax using the K expression builder, which handles the $and construction automatically and often reduces syntax errors [2][5]. Example using K-Expressions: from chromadb import K # This compiles to the same $and JSON structure as above results = collection.get( where=(K("key1") == "value1") & (K("key2") >= 10)) Important Note: Do not attempt to put multiple keys directly inside a single dictionary for the where argument (e.g., where={"key1": "value1", "key2": "value2"}), as this is not the standard way to guarantee logical AND behavior across multiple conditions in older versions and may cause unexpected results or errors [6]. Always use the explicit $and list format or the K-expression builder [1][2][5].
Citations:
- 1: https://docs.trychroma.com/docs/querying-collections/metadata-filtering
- 2: https://docs.trychroma.com/reference/where-filter
- 3: https://github.com/amikos-tech/chroma-cookbook/blob/main/examples/filtering/python/filter_examples.py
- 4: https://chroma-core-chroma.mintlify.app/guides/filtering
- 5: https://docs.trychroma.com/cloud/search-api/filtering
- 6: chroma-core/chroma#825
Use
chunkIdfor thegetPathsfilter and wrap multi-condition filters in$and.
getPathsfilters againstchunk, but these chunks are stored/referenced aschunkId, so the query matches the wrong metadata key. Use{'chunkId': {'$eq': 0}}and wrap it with$andwhen adding theparentcondition.🧰 Tools
🪛 ast-grep (0.44.1)
[info] 377-377: use jsonify instead of json.dumps for JSON output
Context: json.dumps(parent, ensure_ascii=False)
Note: [CWE-116] Improper Encoding or Escaping of Output.(use-jsonify)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_chroma/chroma.py` around lines 374 - 378, Update the getPaths filter construction to use the chunkId metadata key with an equality condition for 0. When parent is provided, combine the chunkId and parent conditions under a $and wrapper, preserving the existing parent serialization.
498-503: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail # Locate and inspect the relevant file and methods without executing repo code. printf 'Files matching chroma.py:\n' fd -a 'chroma\.py$' . || true printf '\nRelevant source excerpts:\n' if [ -f nodes/src/nodes/store_chroma/chroma.py ]; then wc -l nodes/src/nodes/store_chroma/chroma.py sed -n '460,535p' nodes/src/nodes/store_chroma/chroma.py | cat -n fi printf '\nChroma dependency/version references:\n' for f in pyproject.toml requirements.txt requirements-dev.txt setup.py setup.cfg Pipfile poetry.lock uv.lock; do [ -f "$f" ] && { echo "--- $f"; rg -n "chromadb|chroma" "$f" || true; } done printf '\nSearch Chroma update/get usage in repo:\n' rg -n "collectionObj\.(get|update)|\.get\(.*include|include=\['metadatas'\]|update\(ids=|new_metadata" nodes/src/nodes/store_chroma/chroma.pyRepository: rocketride-org/rocketride-server
Length of output: 4343
🌐 Web query:
ChromaDB collection get include metadatas result dict shape ids metadatas💡 Result:
In ChromaDB, the collection.get method returns data in a column-oriented format (a dictionary of lists) [1][2]. The results are returned as a GetResult object [3][4]. When you specify include=["metadatas"] (or include=["documents", "metadatas"], which is the default), the structure of the returned dictionary is as follows [3][5]: - ids: A flat list of strings, representing the IDs of the retrieved records [1][6]. - metadatas: An optional list of dictionaries, where each dictionary corresponds to the metadata of the record at the same index in the ids list [3][1][2]. - documents: An optional list of strings (if included), corresponding to the record at the same index [3][1][2]. Because the data is column-oriented, all returned lists (ids, metadatas, documents, etc.) are aligned by index [1][2]. For example, ids[i], metadatas[i], and documents[i] all refer to the same record [1][2]. Example usage: result = collection.get(include=["documents", "metadatas"]) # Accessing data by index for id, doc, meta in zip(result["ids"], result["documents"], result["metadatas"]): print(id, doc, meta) If a field is not requested in the include parameter, it will not be present in the returned dictionary [3][1]. Note that ids are always returned, regardless of the include parameter [1][2].
Citations:
- 1: https://docs.trychroma.com/docs/querying-collections/query-and-get
- 2: https://cookbook.chromadb.dev/core/collections/
- 3: https://docs.trychroma.com/reference/python/collection
- 4: https://github.com/chroma-core/chroma/blob/b821068bf420257633a7fbacfffe1a2fe66ec45a/chromadb/api/types.py
- 5: https://github.com/chroma-core/chroma/blob/main/chromadb/api/models/Collection.py
- 6: https://github.com/chroma-core/agent-skills/blob/main/skills/chroma-local/querying/python.md
🌐 Web query:
ChromaDB collection update API ids metadatas where new_metadata💡 Result:
In ChromaDB, you update the metadata of existing records using the update method on a collection [1][2]. When calling update, passing the ids parameter along with the new metadata dictionary in the metadatas parameter will overwrite the existing metadata for those specific records [3][4]. Key points regarding the update process: 1. Metadata Replacement: The update operation replaces the entire metadata dictionary for the specified IDs [3][4]. If you wish to retain existing keys and only add or modify specific ones, you must first retrieve the current metadata using get, update the dictionary in your application code, and then pass the complete, modified dictionary back to the update method [5]. 2. Selective Updating: You do not need to provide documents or embeddings when updating metadata [2]. If you only provide ids and metadatas, ChromaDB will update the metadata in place without re-generating embeddings or altering the document content [2]. 3. Upsert Alternative: You can also use the upsert method, which behaves identically to update for existing IDs but will create a new record if the provided ID does not already exist [1][2]. Example usage: # To update metadata without overwriting existing fields: # 1. Get the current metadata record = collection.get(ids=["doc1"], include=["metadatas"]) new_metadata = record["metadatas"] new_metadata["new_key"] = "new_value" # Add or update keys # 2. Update the record collection.update( ids=["doc1"], metadatas=[new_metadata]) Note that if you are looking to update the metadata of the collection itself (as an object) rather than individual records, you must use the collection.modify(metadata={...}) method, which also fully replaces the collection's metadata [6][7][5].
Citations:
- 1: https://docs.trychroma.com/docs/collections/update-data
- 2: https://www.datacamp.com/tutorial/chromadb-tutorial-step-by-step-guide
- 3: https://www.mintlify.com/chroma-core/chroma/concepts/metadata
- 4: https://mintlify.wiki/chroma-core/chroma/concepts/metadata
- 5: https://cookbook.chromadb.dev/core/collections/
- 6: https://www.mintlify.com/chroma-core/chroma/concepts/collections
- 7: https://docs.trychroma.com/docs/collections/manage-collections
Fix soft-delete/reactivate metadata updates.
get(include=['metadatas'])returns columns (ids,metadatas), so both methods currently iterate keys likeids/metadatasand pass unsupportedwhere=/new_metadata=kwargs toupdate(). Build parallelidsandmetadataslists, modifymetadata['isDeleted'], then callself.collectionObj.update(ids=ids, metadatas=metadatas)while preserving existing metadata keys. Mirror this change inmarkActive.📍 Affects 1 file
nodes/src/nodes/store_chroma/chroma.py#L498-L503(this comment)nodes/src/nodes/store_chroma/chroma.py#L522-L527🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_chroma/chroma.py` around lines 498 - 503, The soft-delete and reactivation flows in markDeleted and markActive incorrectly iterate Chroma result keys and call update with unsupported arguments. At both affected sites in nodes/src/nodes/store_chroma/chroma.py (lines 498-503 and 522-527), build parallel ids and metadatas lists from the returned result columns, update each metadata object’s isDeleted value while preserving all existing keys, then call self.collectionObj.update with ids and metadatas.
567-591: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== file locations ==" fd -a 'chroma.py$' . | sed 's#^\./##' echo "== outline ==" ast-grep outline nodes/src/nodes/store_chroma/chroma.py --match render --view expanded || true echo "== relevant source ==" sed -n '530,610p' nodes/src/nodes/store_chroma/chroma.py | cat -n echo "== chroma get/include usages ==" rg -n "collectionObj\.get|get\(.*include=" nodes/src/nodes/store_chroma -S -C 3 || true echo "== imports and setup ==" sed -n '1,80p' nodes/src/nodes/store_chroma/chroma.py | cat -n sed -n '480,570p' nodes/src/nodes/store_chroma/chroma.py | cat -n echo "== python/chroma docs from installed package if available ==" python3 - <<'PY' try: import chromadb print("chromadb", getattr(chromadb, "__version__", None)) from chromadb.config import Settings import inspect print("DocumentAPI.get signature:") print(inspect.signature(chromadb.api.models.Collection.Collection.get)) help(chromadb.api.models.Collection.Collection.get) except Exception as e: print("chromadb probe failed:", type(e).__name__, str(e)) PYRepository: rocketride-org/rocketride-server
Length of output: 15507
🌐 Web query:
ChromaDB Python Collection.get include metadatas documents return format results ids metadatas documents💡 Result:
In ChromaDB, the collection.get method returns results in a column-oriented format [1][2][3]. The results are returned as a GetResult dictionary, which organizes each requested field (such as documents, metadatas, or embeddings) into its own list [1][3][4]. The structure of the returned object is as follows: - ids: Always returned as a list of strings [2][3]. - documents: Returned as a list of strings (or nulls) if requested [2][3][4]. - metadatas: Returned as a list of dictionaries if requested [2][3][4]. - embeddings: Returned as a list of lists if requested [2][3][4]. Crucially, because the format is column-oriented, the data is aligned by index [2][3]. This means that for any index 'i', results['ids'][i], results['documents'][i], and results['metadatas'][i] all refer to the same record [2][3]. You can control which fields are included in this output using the include parameter [1][3]. By default, if the include parameter is not specified, ChromaDB typically returns both "documents" and "metadatas" [1][3]. Because these lists are aligned, it is common practice to process the results by zipping them together [1][2][3]: result = collection.get(include=["documents", "metadatas"]) # Zip aligned columns to process records for id, document, metadata in zip(result["ids"], result["documents"], result["metadatas"]): print(id, document, metadata)
Citations:
- 1: https://docs.trychroma.com/reference/python/collection
- 2: https://cookbook.chromadb.dev/core/collections/
- 3: https://docs.trychroma.com/docs/querying-collections/query-and-get
- 4: https://github.com/chroma-core/chroma/blob/b821068bf420257633a7fbacfffe1a2fe66ec45a/chromadb/api/types.py
🌐 Web query:
site:docs.trychroma.com reference/python/collection get include metadatas documents None💡 Result:
When you perform a.get operation in ChromaDB and receive None for fields like metadatas or documents, it is not an error [1]. This behavior occurs because ChromaDB does not return all available data by default to optimize performance and reduce memory usage [1][2]. To retrieve specific fields, you must explicitly include them using the include parameter [3][2]. How to resolve this: You can specify which fields to return by passing a list of strings to the include parameter in your.get call [3][2]. For example, if you want to include both documents and metadatas, you should structure your code as follows: results = collection.get( include=["documents", "metadatas"]) Key points: 1. Default Behavior: By default, the.get method returns documents and metadatas [3][2]. If you explicitly set include to something else, or if the data was not stored with those fields, they may appear as None in the result [1][2]. 2. Available Fields: Valid strings for the include parameter include "documents", "metadatas", "embeddings", "uris", and "distances" (though "distances" is typically used with.query) [3][2]. 3. IDs are Always Returned: Note that ids are always returned regardless of the include parameter [2]. 4. Troubleshooting: If you find fields are None even when you expect data, ensure that the data was actually stored for those records in the collection and that you are including them in the include list [1][2].
Citations:
- 1: https://docs.trychroma.com/docs/overview/troubleshooting
- 2: https://docs.trychroma.com/docs/querying-collections/query-and-get
- 3: https://docs.trychroma.com/reference/python/collection
🌐 Web query:
site:github.com/chroma-core/chroma GetResult metadatas None includes_metadata💡 Result:
The GetResult object is a TypedDict used by the ChromaDB vector database to structure the output of collection.get and query operations [1][2][3]. The metadatas field within this object is an Optional[List[Metadata]] [1][3]. When the metadatas field appears as None in a GetResult object, it typically signifies one of the following scenarios: 1. Data Not Requested: The include parameter in the get or query operation did not specify "metadatas" as a field to be included in the response [2][4]. In such cases, the field is often returned as None or omitted, depending on the implementation [5][6]. 2. Records Lack Metadata: If "metadatas" was explicitly included in the include parameter, but the returned records do not have associated metadata, the field may return None for those specific entries or the list itself may be None/empty [1][4]. 3. API Transport: When using the ChromaDB FastAPI client, the server may explicitly pass back None for fields that were not included in the request or are absent, which is then mapped to the GetResult object [4]. To verify which fields were included in a specific operation, the GetResult object also contains an included field (e.g., included: Include), which lists the fields that were requested and returned in the payload [1][7][8]. This field was introduced to clarify the status of non-included fields, preventing ambiguity when a field like embeddings or metadatas is returned as None [7][6][8].
Citations:
- 1: https://github.com/chroma-core/chroma/blob/main/chromadb/api/types.py
- 2: https://github.com/chroma-core/chroma/blob/main/chromadb/api/models/Collection.py
- 3: https://github.com/chroma-core/chroma/blob/b821068bf420257633a7fbacfffe1a2fe66ec45a/chromadb/api/types.py
- 4: https://github.com/chroma-core/chroma/blob/main/chromadb/api/fastapi.py
- 5: chroma-core/chroma#300
- 6: chroma-core/chroma#2028
- 7: chroma-core/chroma#2044
- 8: chroma-core/chroma#845
🌐 Web query:
ChromaDB Python get result metadata is None list when include metadatas💡 Result:
If you are seeing None values in your metadatas list when using ChromaDB's get or query methods, it is typically due to one of the following reasons: 1. Records lack metadata: If a specific document in your collection does not have associated metadata, ChromaDB will return None for that entry [1]. In a returned results dictionary, the metadatas field is a list where each index corresponds to the document at the same index in the ids or documents lists [2][3]. If no metadata exists for a record, the corresponding entry in the metadatas list will be None [1]. 2. Include parameter configuration: Ensure that "metadatas" is explicitly included in the include argument of your get or query call [2][4]. While "metadatas" is often included by default, it is best practice to specify it if you are encountering unexpected None results to ensure the data is fetched from the database [2][5]. Example of correct usage: result = collection.get(include=["documents", "metadatas"]) 3. Data insertion issues: Verify that you correctly assigned metadata when adding documents to the collection [6]. If the metadata was not correctly passed (e.g., using an incorrect list/dictionary structure during bulk insertion), it may result in None values later [6]. Note that ChromaDB returns data in a column-oriented format [2][3]. If you have multiple results, ensure you are iterating over the aligned lists (e.g., using zip) to correctly associate IDs, documents, and their respective metadata [2][3]. If you still see None unexpectedly, inspect the raw data added to the collection to confirm that the records were stored with the expected metadata dictionary [6].
Citations:
- 1: chroma-core/chroma#3510
- 2: https://docs.trychroma.com/docs/querying-collections/query-and-get
- 3: https://cookbook.chromadb.dev/core/collections/
- 4: https://docs.trychroma.com/docs/overview/troubleshooting
- 5: https://docs.trychroma.com/reference/python/collection
- 6: chroma-core/chroma#2914
🌐 Web query:
ChromaDB Python httpClient GetResult metadatas None when include metadatas💡 Result:
If you are receiving None for metadatas in ChromaDB when explicitly requesting them via the include parameter in the Python HttpClient, it is likely due to one of the following reasons: 1. Absence of Metadata in Stored Records: If the specific records retrieved do not have any metadata associated with them, ChromaDB will return None (or null) for that specific record in the results list [1]. It is important to note that metadata is optional; if a document was added without metadata, the corresponding entry in the returned list will be None [1]. 2. Incorrect Include Parameter Format: Ensure the include parameter is passed as a list of strings [2][3]. For the get method, the include list should contain "metadatas" [2][4]. Example of correct usage: collection.get( ids=["your_id"], include=["documents", "metadatas"]) 3. Client-Server Deserialization or Version Issues: There have been historical reports of inconsistencies between different client versions (e.g., Python vs. TypeScript/JavaScript) or during API serialization where metadata appears as null [5][6]. If you are using the HttpClient, ensure your client version is up-to-date and matches the version of the Chroma server [5]. 4. Checking the Data Source: Verify that metadata was indeed stored correctly during the initial add or upsert operation [7][5]. You can debug this by checking a single, known record to confirm that metadata exists in the database [1]. If the returned metadatas list contains None values, it confirms that the request was successful but that the specific documents retrieved simply have no metadata stored in the collection [1]. If the entire metadatas field is missing (None instead of a list), double-check that "metadatas" is correctly included in the list passed to the include argument [2][7][4].
Citations:
- 1: chroma-core/chroma#3510
- 2: https://docs.trychroma.com/docs/querying-collections/query-and-get
- 3: https://chroma-core-chroma.mintlify.app/api/python/collection
- 4: https://docs.trychroma.com/reference/python/collection
- 5: chroma-core/chroma#2914
- 6: chroma-core/chroma#4273
- 7: https://stackoverflow.com/questions/76482987/chroma-database-embeddings-none-when-using-get
🌐 Web query:
"def get" "include" "_include_metadatas" "metadatas" chromadb api collection get💡 Result:
In the ChromaDB Python API, the
getmethod on a collection uses theincludeparameter to specify which fields should be returned for the retrieved records [1][2]. Thegetmethod is defined as follows:python collection.get( ids=None, where=None, limit=None, offset=None, where_document=None, include=["metadatas", "documents"] )Key points regarding theincludeparameter: - Purpose: It determines which fields (besidesids, which are always returned) are included in the results [3][4]. - Available options: You can include"embeddings","metadatas","documents", and"uris"[1][2][5]. - Default value: If theincludeparameter is not specified, it defaults to["metadatas", "documents"][1][6]. - Optimization: To minimize response size and improve performance, it is recommended to only include the fields you explicitly need for your application [7][4]. Example usage:python # Retrieve only the documents results = collection.get(include=["documents"]) # Retrieve documents and embeddings results = collection.get(include=["documents", "embeddings"])When usingget, results are returned as aGetResultobject (a dictionary), where each key corresponds to the fields requested in theincludelist [3][2]. These arrays are aligned by index, meaning that the record atids[i]corresponds to the data atdocuments[i],metadatas[i], etc. [3][4].Citations:
- 1: https://github.com/chroma-core/chroma/blob/main/chromadb/api/models/Collection.py
- 2: https://mintlify.wiki/chroma-core/chroma/api/python/collection
- 3: https://docs.trychroma.com/docs/querying-collections/query-and-get
- 4: https://cookbook.chromadb.dev/core/collections/
- 5: https://chroma-core-chroma.mintlify.app/api/python/collection
- 6: https://docs.trychroma.com/reference/python/collection
- 7: https://github.com/chroma-core/agent-skills/blob/main/skills/chroma-local/querying/python.md
🌐 Web query:
"GetResult" "metadatas" "Optional[List[Metadata]]"💡 Result:
The type definition
metadatas: Optional[List[Metadata]]within aGetResultobject is primarily associated with the ChromaDB Python API [1][2][3]. In ChromaDB,GetResultis a TypedDict representing the payload returned bycollection.get()operations [1][4]. The structure typically includes the following fields, wheremetadatasis an optional field containing the metadata for the retrieved records [1][5]: ids: List[ID] embeddings: Optional[Union[Embeddings,...]] documents: Optional[List[Document]] uris: Optional[URIs] data: Optional[Loadable] metadatas: Optional[List[Metadata]] included: Include When usingcollection.get(), you can control the returned fields using theincludeparameter (e.g.,include=["documents", "metadatas"]) [4][5]. ChromaDB returns these results in a columnar (or column-major) format, where corresponding entries across the returned lists (e.g.,ids,documents,metadatas) represent the same record [1][2][4]. Other systems may useGetResultandmetadataterminology, but the specific Python type hintOptional[List[Metadata]]is characteristic of ChromaDB's implementation [1][3][6][7]. For instance, FDC3 usesgetResultMetadata()returningPromise<ContextMetadata>[8][9], and Oracle NoSQL usesGetResultobjects where metadata is accessed via specific getter methods rather than a typed list field [6][10].Citations:
- 1: https://github.com/chroma-core/chroma/blob/main/chromadb/api/types.py
- 2: https://docs.trychroma.com/docs/querying-collections/query-and-get
- 3: https://github.com/chroma-core/chroma/blob/b821068bf420257633a7fbacfffe1a2fe66ec45a/chromadb/api/types.py
- 4: https://docs.trychroma.com/reference/python/collection
- 5: https://chroma-core-chroma.mintlify.app/api/python/collection
- 6: https://docs.oracle.com/en/database/other-databases/nosql-database/25.3/nsdev/using-row-metadata-read-operations.html
- 7: https://github.com/apache/pulsar/blob/master/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/api/MetadataStore.java
- 8: https://fdc3.finos.org/docs/next/api/ref/Metadata
- 9: https://fdc3.finos.org/docs/next/api/spec
- 10: https://docs.oracle.com/en/database/other-databases/nosql-database/25.3/nsdev/using-get-api-fetch-data.html
🌐 Web query:
ChromaDB Python client includes_include_data include data metadata default True💡 Result:
In the ChromaDB Python client, the include parameter is used to specify which optional data fields (such as "metadatas", "documents", "embeddings", "distances", or "uris") should be returned by query and get operations [1][2][3][4]. The IDs of the records are always returned by default, regardless of the include parameter [1][4]. The default behavior for the include parameter is as follows: For get: The default value is ["metadatas", "documents"] [1][2][4]. For query: The default value is ["metadatas", "documents", "distances"] [1][4]. You can override these defaults by explicitly passing an include list to the method call [3][4]. For example: collection.get(include=["documents"]) collection.query(query_texts=["example query"], include=["metadatas", "embeddings"]) There is no parameter named includes_include_data in the ChromaDB Python client API [2][5][6]. The confusion may stem from the include parameter itself, which serves the purpose of defining what data to include in the response [1][4].
Citations:
- 1: https://cookbook.chromadb.dev/core/collections/
- 2: https://github.com/chroma-core/chroma/blob/main/chromadb/api/models/Collection.py
- 3: https://github.com/chroma-core/agent-skills/blob/main/skills/chroma-local/querying/python.md
- 4: https://docs.trychroma.com/docs/querying-collections/query-and-get
- 5: https://docs.trychroma.com/reference/python/collection
- 6: https://chroma-core-chroma.mintlify.app/api/python/collection
Always fetch metadata before reading it in
render.
collection.get(include=['documents'])requests documents only, andGetResult.metadatasis omitted/Nonewhen not requested.results['metadatas'][i]therefore hits one of these unhandled paths:None[i]or an unrequested metadata row. Include"metadatas"in the includes list and access chunk IDs from the returned metadata dict.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_chroma/chroma.py` around lines 567 - 591, Update the collection.get call in render to request both documents and metadatas, then read each row’s chunkId from the returned metadata dictionary before applying the offset and range checks. Preserve the existing text-array population logic and avoid treating metadata as an object attribute.nodes/src/nodes/store_elasticsearch/elasticsearch_store.py (3)
93-116: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash fd -t f 'services*.json' nodes/src/nodes/store_elasticsearch --exec cat {} rg -nP "get\(['\"]mode['\"]" nodes/src/nodes/store_elasticsearchRepository: rocketride-org/rocketride-server
Length of output: 761
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== file sizes ==" wc -l nodes/src/nodes/store_elasticsearch/IGlobal.py nodes/src/nodes/store_elasticsearch/elasticsearch_store.py nodes/src/nodes/store_elasticsearch/configs/*.json 2>/dev/null || true echo "== IGlobal relevant sections ==" sed -n '1,320p' nodes/src/nodes/store_elasticsearch/IGlobal.py echo "== elasticsearch_store relevant sections ==" sed -n '1,180p' nodes/src/nodes/store_elasticsearch/elasticsearch_store.py echo "== config files ==" fd -t f -d 3 . nodes/src/nodes/store_elasticsearch -e json -e py -e md | sort | xargs -r -n 1 printf '\n--- %s ---\n' | xargs -r -n 1 cat echo "== mode references across node ==" rg -n "\bmode\b|get\(.*'mode'|get\(.*\"mode\"|vstore|INDEX|index" nodes/src/nodes/store_elasticsearchRepository: rocketride-org/rocketride-server
Length of output: 20787
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== locate node config/schema-related files ==" git ls-files nodes/src/nodes/store_elasticsearch | sort echo "== config/schema files ==" while IFS= read -r f; do if [ "${f##*.}" = "json" ] || [ "${f##*.}" = "md" ] || [ "${f##*.}" = "yml" ] || [ "${f##*.}" = "yaml" ]; then printf '\n--- %s ---\n' "$f" cat "$f" fi done < <(git ls-files nodes/src/nodes/store_elasticsearch | sort) echo "== constants ==" cat nodes/src/nodes/store_elasticsearch/constants.py echo "== focused mode/config references ==" rg -n "\bmode\b|deploy|self-managed|vector_database|vstore|index_search|Config.getNodeConfig|getConnConfig|validation|schema|property" nodes/src/nodes/store_elasticsearchRepository: rocketride-org/rocketride-server
Length of output: 1099
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== node files ==" git ls-files nodes/src/nodes/store_elasticsearch | sort echo "== services and docs ==" for f in nodes/src/nodes/store_elasticsearch/*.py nodes/src/nodes/store_elasticsearch/services*.json nodes/src/nodes/store_elasticsearch/README.md nodes/src/nodes/store_elasticsearch/elasticsearch/README.md nodes/src/nodes/store_elasticsearch/opensearch/README.md nodes/src/nodes/store_elasticsearch/constants.py; do [ -e "$f" ] || continue printf '\n--- %s ---\n' "$f" sed -n '1,220p' "$f" done echo "== constants full ==" cat nodes/src/nodes/store_elasticsearch/constants.py echo "== focused mode/config references ==" rg -n "\bmode\b|deploy|self-managed|vector_database|vstore|index_search|Config.getNodeConfig|getConnConfig|validation|schema|property" nodes/src/nodes/store_elasticsearchRepository: rocketride-org/rocketride-server
Length of output: 50389
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== elasticsearch schema tail ==" wc -l nodes/src/nodes/store_elasticsearch/services.elasticsearch.json sed -n '1,220p' nodes/src/nodes/store_elasticsearch/services.elasticsearch.json sed -n '220,420p' nodes/src/nodes/store_elasticsearch/services.elasticsearch.json sed -n '420,620p' nodes/src/nodes/store_elasticsearch/services.elasticsearch.json sed -n '620,840p' nodes/src/nodes/store_elasticsearch/services.elasticsearch.json echo "== README snippets ==" rg -n "mode|deploy|local|cloud|vstore|index" nodes/src/nodes/store_elasticsearch/README.md nodes/src/nodes/store_elasticsearch/elasticsearch/README.md echo "== config schema/source evidence: whether mode is separate deployment field ==" python3 - <<'PY' import json, pathlib for p in pathlib.Path('nodes/src/nodes/store_elasticsearch').glob('*.json'): data=json.loads(p.read_text()) print(f'--- {p}') props=data.get('fields',{}) for k,v in props.items(): if k=='elasticsearch.mode': print(f'ELASTICSEARCH MODE: {json.dumps(v, indent=2)[:1200]}') # inspect service preconfig mode values that exist in git for p in pathlib.Path('nodes/src/nodes/store_elasticsearch').glob('services*.json'): print(f'--- {p} preconfig mode values') for name,prof in data.get('preconfig',{}).get('profiles',{}).items(): print(name, prof.get('mode')) PYRepository: rocketride-org/rocketride-server
Length of output: 22773
Resolve Elasticsearch store deployment mode during config mapping.
Store.__init__derives the Elasticsearch URL scheme fromconfig.get('mode'), but the node service defines deployment profiles with that same key and a separateelasticsearch.profiledeployment field. A local profile withmode: cloud-hostedwould usefalsestore mode while constructinghttps://localhost:..., diverging from the documented/local scheme behavior; the store URL host scheme belongs in the deployment/profile logic, not themodekey.🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 109-109: Do not make http calls without encryption
Context: 'http://'
Note: [CWE-319] Cleartext Transmission of Sensitive Information.(requests-http)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_elasticsearch/elasticsearch_store.py` around lines 93 - 116, Update the Elasticsearch configuration mapping before Store.__init__ so deployment mode is resolved from the Elasticsearch deployment/profile field, not config.mode. Pass the resolved profile-based scheme or mode into Store, and ensure Store.__init__ uses that deployment value when selecting the default URL scheme while preserving explicit http/https host schemes.
382-383: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Raise
Exception, notBaseException.
BaseExceptionbypasses normalexcept Exceptionhandlers (and is the base ofSystemExit/KeyboardInterrupt), so this can escape the pipeline's error handling unexpectedly.🐛 Proposed fix
- if docFilter.offset: - raise BaseException('Non-zero offset is not supported in semantic searching') + if docFilter.offset: + raise Exception('Non-zero offset is not supported in semantic searching')📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.if docFilter.offset: raise Exception('Non-zero offset is not supported in semantic searching')🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_elasticsearch/elasticsearch_store.py` around lines 382 - 383, Update the offset validation in the Elasticsearch store’s semantic-search flow to raise Exception instead of BaseException, preserving the existing error message and condition for non-zero docFilter.offset values.
583-591: 🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win
Payload-size batching is ineffective;
sys.getsizeof(doc)only measures the dict header.
sys.getsizeofis shallow — it does not count the embedding list, content, or metadata referenced bydoc, sosum_sizestays tiny andsum_size > self.payload_limitessentially never fires. Only the hardcoded500count bound is active. Estimate the serialized size (e.g.len(json.dumps(doc['_source']))or embedding length × item size) if the payload guard is meant to protect against large bulk requests. The literal500could also reuseDEFAULT_TEXT_BATCH_SIZE.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_elasticsearch/elasticsearch_store.py` around lines 583 - 591, The payload batching logic in the bulk document loop must measure serialized document content rather than the shallow dict size from sys.getsizeof(doc). Update the size calculation near _run_bulk to use the serialized _source (or equivalent payload-size estimate), and reuse DEFAULT_TEXT_BATCH_SIZE instead of the hardcoded 500 when appropriate; preserve flushing when either count or payload limits are reached.nodes/src/nodes/store_elasticsearch/README.md (3)
37-75: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Fix the Elasticsearch configuration table.
This section mixes OpenSearch fields (
host,enabled,username,password,collection) and even listsprovideras"opensearch"under the Elasticsearch heading. That contradicts the generatedservices.elasticsearch.jsonschema and the Elasticsearch loader, which use profile/vector fields andprovider: "elasticsearch". Users following this table will configure the wrong keys.🧰 Tools
🪛 LanguageTool
[style] ~50-~50: Consider removing “of” to be more concise
Context: ...rms, 'and' matches documents containing ALL of the query terms, 'exact' requires the exact...(ALL_OF_THE)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_elasticsearch/README.md` around lines 37 - 75, Replace the outdated Elasticsearch configuration table with entries matching the services.elasticsearch.json schema and Elasticsearch loader: use the profile/vector fields shown in the lower table, remove OpenSearch-only keys such as host, enabled, username, password, and collection, and set provider to "elasticsearch" where applicable. Keep the documented defaults and profile-specific fields accurate.
124-126: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== locate files ==" fd -a 'README.md|store_elasticsearch|opensearch|elasticsearch|node' nodes . | sed 's#^\./##' | head -100 echo "== target README section ==" if [ -f nodes/src/nodes/store_elasticsearch/README.md ]; then nl -ba nodes/src/nodes/store_elasticsearch/README.md | sed -n '1,220p' fi echo "== search references ==" rg -n "verify_certs|opensearch|OpenSearch|elastic|store_elasticsearch|http://.*https|self-signed|certificate" nodes/src/nodes nodes -S --glob '!**/*.map' | head -200Repository: rocketride-org/rocketride-server
Length of output: 5986
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== target README ==" if [ -f nodes/src/nodes/store_elasticsearch/README.md ]; then nl -ba nodes/src/nodes/store_elasticsearch/README.md | sed -n '1,220p' else echo "missing target README" fi echo "== store_elasticsearch files ==" find nodes/src/nodes/store_elasticsearch -maxdepth 4 -type f -print | sort echo "== OpenSearch/verify_certs/elasticsearch references ==" rg -n -C 3 "verify_certs|opensearch|OpenSearch|http://.*https|self-signed|certificate|auth" nodes/src/nodes/store_elasticsearch nodes/src/nodes -S | head -250Repository: rocketride-org/rocketride-server
Length of output: 230
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== target README ==" if [ -f nodes/src/nodes/store_elasticsearch/README.md ]; then awk '{printf "%6d\t%s\n", NR, $0}' nodes/src/nodes/store_elasticsearch/README.md | sed -n '1,220p' else echo "missing target README" fi echo "== store_elasticsearch files ==" find nodes/src/nodes/store_elasticsearch -maxdepth 4 -type f -print | sort echo "== OpenSearch/verify_certs/elasticsearch references ==" rg -n -C 3 "verify_certs|opensearch|OpenSearch|http://.*https|self-signed|certificate|auth" nodes/src/nodes/store_elasticsearch nodes/src/nodes -S | head -300Repository: rocketride-org/rocketride-server
Length of output: 47710
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== opensearch client outline ==" if [ -f nodes/src/nodes/store_elasticsearch/opensearch_client.py ]; then awk '{printf "%6d\t%s\n", NR, $0}' nodes/src/nodes/store_elasticsearch/opensearch_client.py | sed -n '1,240p' fi echo "== IGlobal init relevant sections ==" if [ -f nodes/src/nodes/store_elasticsearch/IGlobal.py ]; then awk '{printf "%6d\t%s\n", NR, $0}' nodes/src/nodes/store_elasticsearch/IGlobal.py fiRepository: rocketride-org/rocketride-server
Length of output: 28980
Do not describe disabled TLS verification as suitable for production.
IGlobal._begin_opensearch()initializes OpenSearch withverify_certs=Falseandssl_show_warn=False; when basic auth is enabled, credentials are sent over a connection whose server certificate is not validated, so this is an insecure compatibility fallback, not a safe default. Update the OpenSearch docs to make the insecure fallback explicit and add/configure a CA verification option before using production deployments with self-signed certificates.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_elasticsearch/README.md` around lines 124 - 126, Update the OpenSearch “Basic auth” documentation to describe verify_certs=False and ssl_show_warn=False as an insecure compatibility fallback, not suitable for production. Explicitly instruct production users with self-signed certificates to configure CA certificate verification before deployment, while preserving the existing authentication requirements.
187-187: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Regenerate the source link after the package move.
The generated README still links to the old
nodes/src/nodes/index_searchpath, while the package and service path are now undernodes/src/nodes/store_elasticsearch. Update the generator/source metadata and rerunnodes:docs-generate; do not hand-edit the generated block.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_elasticsearch/README.md` at line 187, Update the source metadata used by the README generator for the Elasticsearch node so its generated source link targets nodes/src/nodes/store_elasticsearch instead of nodes/src/nodes/index_search. Then rerun nodes:docs-generate to regenerate the README block; do not edit the generated link manually.Source: Coding guidelines
nodes/src/nodes/store_milvus/milvus.py (7)
104-109: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== files ==" git ls-files | rg '(^|/)milvus\.py$|(^|/)ROCKETRIDE_python_API\.md$|config\.json$' || true echo "== target snippet and surrounding relevant lines ==" sed -n '1,240p' nodes/src/nodes/store_milvus/milvus.py | cat -n echo "== searches for similarity/config usage ==" rg -n "similarity|FLOAT_VECTOR|create_collection|insert_index|index" nodes/src/nodes/store_milvus nodes -g '*.py' -g '*.json' || trueRepository: rocketride-org/rocketride-server
Length of output: 50389
🌐 Web query:
Milvus documentation FLOAT_VECTOR index metric_type L2 IP COSINE JACCARD HAMMING BM25💡 Result:
In Milvus, supported metric types depend on the vector field data type [1]. For fields of type FLOAT_VECTOR, the supported similarity metrics are COSINE, L2, and IP [1][2]. The metric types you mentioned are categorized based on the vector field type as follows: FLOAT_VECTOR (and other dense floating-point vector types like FLOAT16_VECTOR, BFLOAT16_VECTOR, INT8_VECTOR): Supported metrics are COSINE, L2, and IP [1]. BINARY_VECTOR: Supported metrics are JACCARD and HAMMING [1][3]. SPARSE_FLOAT_VECTOR: Supported metrics are IP and BM25 [1]. Note that BM25 is specifically designed for full-text search and is used exclusively with sparse vectors (SPARSE_FLOAT_VECTOR), not with FLOAT_VECTOR [1][4][5]. When using BM25, the required index type is typically SPARSE_INVERTED_INDEX [6][4].
Citations:
- 1: https://milvus.io/docs/v2.6.x/metric.md
- 2: https://milvus.io/docs/v2.5.x/metric.md
- 3: https://milvus.io/docs/index.md
- 4: https://milvus.io/docs/full-text-search.md
- 5: https://milvus.io/docs/bm25-function.md
- 6: https://milvus.io/docs/v2.6.x/index-vector-fields.md
Restrict Milvus float-vector similarity metrics to supported values.
vectoris aFLOAT_VECTOR, but config valuesJACCARD,HAMMING, andBM25pass this validation even though MILVUS only supportsCOSINE,L2, andIPfor float vectors (JACCARD/HAMMINGare for binary vectors;BM25is for sparse vectors). These values would be accepted and then fail collection/index creation because they are invalid for this schema.🐛 Proposed fix
- if similarity in ['L2', 'IP', 'COSINE', 'JACCARD', 'HAMMING', 'BM25']: + if similarity in ['L2', 'IP', 'COSINE']:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.# check if the similarity matches milvus configuration options similarity = config.get('similarity', 'COSINE') if similarity in ['L2', 'IP', 'COSINE']: self.similarity = similarity else: raise Exception('The metric you provided in the config.json does not match required milvus configurations')🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_milvus/milvus.py` around lines 104 - 109, Update the similarity validation in the Milvus initialization logic to accept only COSINE, L2, and IP for the FLOAT_VECTOR schema. Remove JACCARD, HAMMING, and BM25 from the allowed values while preserving the existing exception for unsupported metrics.
205-207: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== locate file ==" fd -a 'milvus.py$' . || true echo "== file excerpt around relevant lines ==" wc -l nodes/src/nodes/store_milvus/milvus.py sed -n '1,260p' nodes/src/nodes/store_milvus/milvus.py | cat -n echo "== search for docFilter isTable stores ==" rg -n "isTable|isDeleted|meta\[" nodes/src/nodes/store_milvus/milvus.py . --glob '*.py' || trueRepository: rocketride-org/rocketride-server
Length of output: 49987
🌐 Web query:
Milvus JSON boolean filter BooleanExpression isTable == bool string literal💡 Result:
In Milvus, to filter on a boolean value within a JSON field, you use the standard comparison operator
==with a boolean literal [1]. The boolean literal should be written astrueorfalse(without quotes) [1]. If you have a JSON field namedmetadataand want to filter where the keyisTableis true, your filter expression should look like this: filter = 'metadata["isTable"] == true' Key points for implementation: - Boolean Literals: Usetrueorfalsedirectly. Do not wrap them in quotes, as wrapping them in quotes would treat them as string literals (e.g.,"true") rather than boolean values [2][1]. - JSON Path Syntax: Use bracket notation (e.g.,field["key"]) to access keys within a JSON field [1][3]. - Indexing: For optimal performance, ensure that if you have created an index on the JSON field, thejson_cast_typeis set tobool[1]. Filtering with a different type than the indexedjson_cast_typemay cause the system to bypass the index and perform a slower brute-force scan [1].Citations:
- 1: https://milvus.io/docs/use-json-fields.md
- 2: https://milvus.io/docs/boolean.md
- 3: https://milvus.io/docs/v2.6.x/basic-operators.md
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== local Milvus filter tests around boolean expressions ==" sed -n '1,260p' nodes/test/milvus/test_filter_injection.py | cat -n rg -n "isTable|bool|false|false|True|False" nodes/test/milvus nodes/src/nodes/store_milvus --glob '*.py' || true echo "== Milvus filter behavior probe from mock expression matching ==" sed -n '260,720p' nodes/test/mocks/pymilvus/__init__.py | cat -nRepository: rocketride-org/rocketride-server
Length of output: 27397
Use unquoted boolean literals for Milvus JSON boolean filters.
_escape_milvus_str(docFilter.isTable)convertsTrue/Falseto'True'/'False'and wraps them in quotes, producing malformed Milvus filter expressions. Comparemeta['isTable']with the lowercase boolean literals used by Milvus, e.g.meta['isTable'] == true/meta['isTable'] == false, or build the boolean filter separately. The current(...,)wrapper also makesappend()return a tuple; drop it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_milvus/milvus.py` around lines 205 - 207, Update the docFilter.isTable condition to append a Milvus filter using lowercase, unquoted boolean literals derived from the boolean value, rather than _escape_milvus_str. Remove the extraneous tuple wrapper around must_conditions.append so the condition is appended directly.
417-431: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Inconsistent escaping mechanism:
json.dumps()here vs._escape_milvus_str()everywhere else.
getPaths()(line 431) and the delete filter inaddChunks()(line 473) build filter literals withjson.dumps(), while every other filter-construction site in this file uses the dedicated, unit-tested_escape_milvus_str()helper (seetest_filter_injection.py). Both approaches likely produce safe output, but consolidating onto one tested escaping path reduces the chance of a future edge case slipping through only one of the two mechanisms.🧰 Tools
🪛 ast-grep (0.44.1)
[info] 430-430: use jsonify instead of json.dumps for JSON output
Context: json.dumps(parent)
Note: [CWE-116] Improper Encoding or Escaping of Output.(use-jsonify)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_milvus/milvus.py` around lines 417 - 431, Replace the json.dumps-based escaping in getPaths and the delete filter within addChunks with the existing _escape_milvus_str helper. Preserve the current filter semantics while routing both parent values through the shared, tested escaping mechanism.
450-520: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win
Existing chunks are deleted before validating that all new chunks have embeddings — data loss on partial failure.
addChunksdeletes all existing records for the incomingobjectIds(lines 472-478) before the per-chunk loop that validatesembedding is not None(lines 500-502) and performs the upsert. If any chunk in the batch lacks an embedding, the exception aborts the method after the old data has already been deleted but before any (or all) new chunks are inserted, permanently losing the previous version of the document with no rollback.Move the embedding validation ahead of the delete so a bad chunk aborts before any destructive change is made.
🧰 Tools
🪛 ast-grep (0.44.1)
[info] 472-472: use jsonify instead of json.dumps for JSON output
Context: json.dumps(k)
Note: [CWE-116] Improper Encoding or Escaping of Output.(use-jsonify)
[info] 505-505: use secrets package over random package
Context: random.getrandbits(27)
Note: [CWE-330] Use of Insufficiently Random Values.(avoid-random-python)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_milvus/milvus.py` around lines 450 - 520, In addChunks, validate that every chunk has a non-None embedding before building objectIds or executing the existing deletion via self.client.delete. Move the embedding check out of the insertion loop into a preliminary validation pass, while preserving the current exception behavior and upsert flow for valid batches.
663-666: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
render()readspoint['chunkId']directly, but the query only requestsmetaandcontent— this willKeyError.The query at line 652-654 uses
output_fields=['meta', 'content'];chunkIdis nested inside themetaJSON blob, not a top-level key (as correctly handled ingetPaths()viarecord.get('meta', {})and_convertToDocs()viaentity.get('meta')). As written,render()will raiseKeyError: 'chunkId'on the first non-empty result, breaking document rendering entirely for this store.🐛 Proposed fix
for point in results: content = point['content'] - chunk = point['chunkId'] + chunk = point['meta']['chunkId'] index = chunk - offset📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.for point in results: content = point['content'] chunk = point['meta']['chunkId'] index = chunk - offset🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_milvus/milvus.py` around lines 663 - 666, Update render() to read chunkId from the result’s meta JSON/object, matching the existing getPaths() and _convertToDocs() handling, instead of accessing point['chunkId'] directly. Preserve the current chunk-offset calculation and rendering behavior after extracting the nested value.
144-193: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "Files:" git ls-files | rg '(^|/)milvus\.py$|test_addchunks_return\.py$|Store.*\.py$|DocumentStore' | head -200 echo echo "milvus outline:" ast-grep outline nodes/src/nodes/store_milvus/milvus.py --match Milvus --view expanded || true echo echo "Relevant milvus lines 80-220:" sed -n '80,220p' nodes/src/nodes/store_milvus/milvus.py | nl -ba -v80 echo echo "Relevant milvus lines 440-470:" sed -n '440,470p' nodes/src/nodes/store_milvus/milvus.py | nl -ba -v440 echo echo "Search createCollection definitions/usages:" rg -n "def createCollection|def _createCollection|_createCollection|createCollection\\(" nodes/src -SRepository: rocketride-org/rocketride-server
Length of output: 401
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "Relevant milvus lines 80-220:" sed -n '80,220p' nodes/src/nodes/store_milvus/milvus.py | awk '{printf "%4d\t%s\n", NR+79, $0}' echo echo "Relevant milvus lines 440-470:" sed -n '440,470p' nodes/src/nodes/store_milvus/milvus.py | awk '{printf "%4d\t%s\n", NR+439, $0}' echo echo "Search createCollection definitions/usages:" rg -n "def createCollection|def _createCollection|_createCollection|createCollection\\(" nodes/src -S echo echo "Test addchunks return relevant lines:" if [ -f nodes/test/milvus/test_addchunks_return.py ]; then awk '{printf "%4d\t%s\n", NR, $0}' nodes/test/milvus/test_addchunks_return.py fi echo echo "Find DocumentStore parent files:" rg -n "class .*Milvus|DocumentStore|createCollection\b" nodes/src/nodes/store_milvus/milvus.py -SRepository: rocketride-org/rocketride-server
Length of output: 11777
🌐 Web query:
Milvus Python client DataType max_length VARCHAR primary key max_length documentation💡 Result:
When using the Milvus Python client (pymilvus), a VARCHAR field used as a primary key must have a defined max_length property [1][2]. The value of max_length defines the maximum byte length for strings allowed to be inserted into that field [1][2]. Key details regarding VARCHAR max_length in Milvus: 1. Valid Range: The permitted value range for max_length is [1] [1][2][3]. 2. Mandatory Requirement: Setting max_length is mandatory when defining a field with DataType.VARCHAR [1][2][3]. 3. Byte vs. Character Length: The max_length parameter specifies the maximum number of bytes, not the number of characters [1][2]. Because multibyte characters (such as certain Unicode/UTF-8 characters) may occupy more than one byte each, you should ensure that the byte length of your strings does not exceed the configured limit [1][2]. 4. Immutability: Once a collection is created, the max_length of a VARCHAR field cannot be changed [4]. 5. Internal Logic: The max_length value is used by Milvus for internal operations, such as allocating memory for insert buffers, so it is recommended to choose an accurate value rather than always defaulting to the maximum unless necessary [4]. Example of defining a VARCHAR primary key in your schema: from pymilvus import DataType schema.add_field( field_name="your_pk_field", datatype=DataType.VARCHAR, is_primary=True, auto_id=False, # Set to True if you want Milvus to generate IDs instead max_length=100 # Required for VARCHAR; range [1])
Citations:
- 1: https://milvus.io/api-reference/pymilvus/v2.6.x/MilvusClient/CollectionSchema/add_field.md
- 2: https://milvus.io/api-reference/pymilvus/v2.5.x/MilvusClient/CollectionSchema/add_field.md
- 3: https://milvus.io/docs/string.md
- 4: milvus-io/milvus#34478
Correct
_createCollection’s success/failure return contract.
_createCollectionreturnsFalsewhen a new collection is created successfully, butaddChunks()checksnot self.createCollection(chunks)and skips all writes onFalse; creation exceptions are currently reported asTrue(success). Adjust it so successful collection creation/existence returnsTrueand creation failures returnFalse.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_milvus/milvus.py` around lines 144 - 193, The _createCollection method currently reverses the expected success contract. Update its return values so an existing collection and successful create_collection call return True, while exceptions during creation return False; preserve the existing collection-existence check and deadlock-avoidance behavior.
156-156: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== repo files matching milvus.py ==" fd -a 'milvus.py$' . | sed 's#^\./##' echo "== target excerpt ==" cat -n nodes/src/nodes/store_milvus/milvus.py | sed -n '130,175p' echo "== pymilvus usages and imports ==" rg -n "from pymilvus|import pymilvus|max_length|DataType\.INT64|schema\.add_field" -S . echo "== python/pymilvus availability ==" python3 - <<'PY' try: import pymilvus print("pymilvus", getattr(pymilvus, "__version__", "unknown")) from pymilvus import DataType print("DataType.INT64", DataType.INT64) print("DataType.VARCHAR", DataType.VARCHAR) except Exception as e: print("pymilvus unavailable:", type(e).__name__, str(e)) PYRepository: rocketride-org/rocketride-server
Length of output: 7746
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== pyproject/package metadata references ==" rg -n 'pymilvus|MilvusClient|store_milvus|nodes/src/nodes/store_milvus' -S pyproject.toml requirements*.txt setup.py setup.cfg poetry.lock uv.lock packages manifest* nodes 2>/dev/null | head -200 echo "== tests/mock schema behavior ==" cat -n nodes/src/nodes/store_milvus/milvus.py | sed -n '1,220p' echo cat -n test/nodes/connectors/milvus/store/test_milvus.py | sed -n '1,140p' echo cat -n nodes/test/mocks/pymilvus/__init__.py | sed -n '120,190p'Repository: rocketride-org/rocketride-server
Length of output: 5571
🌐 Web query:
pymilvus MilvusClient create_schema add_field max_length DataType.INT64 max_length ignored mandatory VARCHAR documentation💡 Result:
In the PyMilvus SDK, the max_length parameter is strictly reserved for string-based data types, specifically DataType.VARCHAR [1][2][3]. When adding a field to a collection schema using add_field, max_length is mandatory for DataType.VARCHAR fields to define the maximum byte length of strings (valid range: 1 to 65,535) [1][2][4][3]. For other data types, such as DataType.INT64, the max_length parameter is not applicable [1][2]. If provided, it is typically ignored or considered invalid for non-string fields [1][2]. Key takeaways regarding your query: 1. Mandatory vs. Ignored: The max_length parameter is mandatory for VARCHAR fields [1][2][5]. If you attempt to use it with DataType.INT64, it serves no function and is ignored [1][2]. 2. Documentation: Official documentation consistently specifies that max_length is required for VARCHAR [1][2][3]. It does not list max_length as a valid parameter for numeric types like INT64 [1][2][6]. 3. Best Practice: To avoid confusion or potential schema validation warnings, you should omit the max_length parameter entirely when defining non-string fields such as INT64 [1][2]. For further details, refer to the official PyMilvus API reference for CollectionSchema.add_field [1][2] or the general schema management documentation [6][5].
Citations:
- 1: https://milvus.io/api-reference/pymilvus/v2.6.x/MilvusClient/CollectionSchema/add_field.md
- 2: https://milvus.io/api-reference/pymilvus/v3.0.x/ORM/CollectionSchema/add_field.md
- 3: https://milvus.io/api-reference/pymilvus/v3.0.x/MilvusClient/CollectionSchema/add_field.md
- 4: https://milvus.io/docs/v2.4.x/schema.md
- 5: https://milvus.io/docs/string.md
- 6: https://milvus.io/docs/schema.md
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== constraints.lock pymilvus entries ==" sed -n '940,1235p' nodes/src/nodes/constraints.lock | rg -n 'pymilvus|Package|Version|Metadata|pydantic|protobuf|grpc|milvus' -C 2 || true echo "== pyproject constraints files ==" fd -a 'pyproject.toml|constraints.lock|requirements.*|setup.*|poetry.lock|uv.lock' nodes/src/nodes | sed 's#^\./##'Repository: rocketride-org/rocketride-server
Length of output: 9302
🌐 Web query:
pymilvus 3.0.0 source CollectionSchema add_field max_length DataType.INT64 validation💡 Result:
In PyMilvus 3.0.0, the
max_lengthparameter is not applicable toDataType.INT64fields. Themax_lengthparameter is specifically mandatory and exclusively used forDataType.VARCHARfields to define the maximum byte length of string data [1][2][3]. ForDataType.INT64fields: - They are scalar fields used for storing 64-bit integers [4]. - They do not supportmax_lengthbecause the size of an integer is fixed by the data type itself [4]. - If you are configuring a field withDataType.INT64, you should not include themax_lengthparameter, as it is intended for string-based validation [1][2]. Validation formax_lengthis strictly enforced forVARCHARfields, with a valid range of [1] bytes [1][3]. Attempting to applymax_lengthto anINT64field would be technically incorrect according to the Milvus schema design [1][2].Citations:
- 1: https://milvus.io/api-reference/pymilvus/v3.0.x/ORM/CollectionSchema/add_field.md
- 2: https://milvus.io/api-reference/pymilvus/v3.0.x/MilvusClient/CollectionSchema/add_field.md
- 3: https://milvus.io/docs/string.md
- 4: https://milvus.io/docs/number.md
🌐 Web query:
"MilvusClient.create_schema" "add_field" "max_length" "DataType.INT64"💡 Result:
In Milvus, the
max_lengthparameter is not used withDataType.INT64[1][2][3]. Themax_lengthparameter is specifically used forDataType.VARCHARfields to define the maximum number of bytes (ranging from 1 to 65,535) that the string field can store [4][5][6]. ForDataType.INT64fields, Milvus handles the storage automatically as a 64-bit integer, and no length or dimension parameters are required [2][3]. If you are adding anINT64field to your schema, you should omitmax_length[5][7]. Example of correctly adding anINT64field: schema.add_field(field_name="my_int64_field", datatype=DataType.INT64) [2][5]Citations:
- 1: https://milvus.io/api-reference/pymilvus/v3.0.x/MilvusClient/Collections/create_schema.md
- 2: https://milvus.io/docs/schema.md
- 3: https://milvus.io/docs/number.md
- 4: https://milvus.io/docs/string.md
- 5: https://milvus.io/api-reference/pymilvus/v3.0.x/MilvusClient/CollectionSchema/add_field.md
- 6: https://milvus.io/docs/v2.4.x/schema.md
- 7: https://milvus.io/api-reference/pymilvus/v3.0.x/ORM/CollectionSchema/add_field.md
🌐 Web query:
site:github.com Milvus-ORM pymilvus add_field max_length DataType.INT64💡 Result:
In Milvus and the Pymilvus ORM, the
max_lengthparameter is not applicable toDataType.INT64[1]. Themax_lengthproperty is specifically used forDataType.VARCHARfields to define the maximum character length of the string [2][1].DataType.INT64fields in Milvus are 64-bit integers and have a fixed size; they do not require or support length-related parameters [3]. When defining a schema usingadd_fieldorFieldSchema, you should only providemax_lengthwhen the data type isDataType.VARCHAR[1][4]. Providingmax_lengthfor anINT64field is not a supported configuration [1].Citations:
- 1: milvus-io/milvus#19090
- 2: https://github.com/milvus-io/pymilvus/blob/6625af7fd0bfdf75fd596e7a9a871528cb539ef7/pymilvus/orm/schema.py
- 3: https://github.com/milvus-io/milvus/blob/master/docs/developer_guides/chap02_schema.md
- 4: milvus-io/milvus#28103
Remove the irrelevant
max_lengthfrom theINT64schema field.
max_lengthapplies toVARCHARfields; on theDataType.INT64primary key this is misleading configuration and should be omitted.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_milvus/milvus.py` at line 156, Remove the max_length argument from the INT64 primary-key field definition in schema.add_field, while preserving the existing field_name, datatype, and is_primary settings.nodes/src/nodes/store_pinecone/pinecone.py (3)
68-68: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
Score threshold never applied in
searchSemantic— attribute name typo.
threshhold_search(line 68, extra “h”) is never assigned;__init__setsself.threshold_search(line 90, correct spelling) instead. Line 290 filters againstself.threshhold_search, which stays at its class-level default0.0forever, so the configuredscorethreshold is silently ignored for semantic search — every match passes the filter regardless of similarity.🐛 Proposed fix
- threshhold_search: float = 0.0 + threshold_search: float = 0.0 @@ - points_satisfy_threshold = [point for point in points if point['score'] >= self.threshhold_search] + points_satisfy_threshold = [point for point in points if point['score'] >= self.threshold_search]Also applies to: 90-90, 289-290
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_pinecone/pinecone.py` at line 68, Correct the misspelled class attribute threshhold_search to threshold_search and update the searchSemantic filtering reference accordingly, matching the self.threshold_search assignment in __init__. Ensure semantic search uses the configured score threshold instead of the class-level default.
226-249: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
searchKeywordcrashes: passes a raw Pinecone filter dict intoget(), which re-converts it as aDocFilter.
searchKeywordbuildsfiltervia_convertFilter(docFilter)(already a Pinecone{'$and': [...]}dict), appends the$containsclause, then callsself.get(filter). Butget()'s parameter is typed/used as aDocFilterand immediately doesself._convertFilter(docFilter=docFilter)— which accessesdocFilter.isDeleted,.nodeId, etc. as attributes. Since a plaindicthas none of these, this raisesAttributeErroron every call tosearchKeyword.🐛 Proposed fix: split filter-application from filter-conversion
- def get(self, docFilter: DocFilter, checkCollection: bool = True) -> List[Doc]: - """ - Given a filter, return the document groups matching the filter. - """ - # If the collection does not exists, by definition there are - # no documents matching the get - if checkCollection and not self.doesCollectionExist(): - return [] - - # Build up the filter - filter = self._convertFilter(docFilter=docFilter) - - # Getting index - index = self.client.Index(self.collection) - vector_size = int(index.describe_index_stats()['dimension']) - - # Perform the search - records = index.query( - vector=[1] * vector_size, top_k=10000, filter=filter, include_metadata=True, include_values=False - )['matches'] - - # Convert the points into documents - docs = self._convertToDocs(records) - return docs + def get(self, docFilter: DocFilter, checkCollection: bool = True) -> List[Doc]: + """ + Given a filter, return the document groups matching the filter. + """ + if checkCollection and not self.doesCollectionExist(): + return [] + filter = self._convertFilter(docFilter=docFilter) + return self._queryFilter(filter) + + def _queryFilter(self, filter: dict) -> List[Doc]: + """Run a pre-built Pinecone filter and convert matches to Docs.""" + index = self.client.Index(self.collection) + vector_size = int(index.describe_index_stats()['dimension']) + records = index.query( + vector=[1] * vector_size, top_k=10000, filter=filter, include_metadata=True, include_values=False + )['matches'] + return self._convertToDocs(records)- # Convert the points into groups - docs = self.get(filter) - - # Return them - return docs + # Convert the points into groups + return self._queryFilter(filter)Also applies to: 297-320
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_pinecone/pinecone.py` around lines 226 - 249, Update searchKeyword and the related get flow so the already-converted Pinecone filter is not passed back through _convertFilter as a DocFilter. Separate filter conversion from filter application, or otherwise reuse the converted filter directly, while preserving the existing DocFilter conversion behavior for normal get callers.
399-468: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
addChunksdeletes all pre-existing chunks for an object before all new batches finish upserting.
objectIdsis collected once for the entirechunksargument (lines 424-426), butself.remove(list(objectIds.keys()))only runs insideflush()on whichever call happens to be first (after whichobjectIdsis cleared). This means the old data for a document is fully deleted before later batches of the same re-ingest have been written. If the pipeline fails/crashes between the first and lastflush(), the old chunks are gone and only a partial set of new chunks exists — permanent data loss for thatobjectId.Since Pinecone has no cross-batch transaction, upsert all new chunks first and only delete stale ones once every batch has succeeded, so a mid-way failure leaves both old and new data present (self-healing on retry) instead of losing data.
🛡️ Proposed fix: defer deletion until after all upserts complete
def flush(): nonlocal points_to_upsert - nonlocal objectIds - - # Getting index index = self.client.Index(self.collection) - - # Batch operation for deletion - if len(objectIds): - self.remove(list(objectIds.keys())) - - # Batch operation for upsert if len(points_to_upsert): index.upsert(vectors=points_to_upsert) - - # Clear them - objectIds = {} points_to_upsert = [] @@ - flush() + flush() + if len(objectIds): + self.remove(list(objectIds.keys()))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.points_to_upsert = [] # Clear the object id list objectIds: Dict = {} def flush(): nonlocal points_to_upsert # Getting index index = self.client.Index(self.collection) # Batch operation for upsert if len(points_to_upsert): index.upsert(vectors=points_to_upsert) # Clear them points_to_upsert = [] # For each document set objectId to true to make sure we delete any chunks we already have added for chunk in chunks: if chunk.metadata.objectId != 'schema': objectIds[chunk.metadata.objectId] = True sum_size = 0 cur_size = 0 # For each document for chunk in chunks: # Get the embedding embedding = chunk.embedding # If we do not have an embedding if embedding is None: raise Exception('No embedding in document') metadata = dict(chunk.metadata) metadata['content'] = chunk.page_content # Bogus document which is not compatible with pinecone because it can't accept all zeroes if chunk.metadata.objectId == 'schema': embedding[-1] = 1 metadata['nodeId'] = '' metadata['parent'] = '' metadata['permissionId'] = 0 metadata['isTable'] = False metadata['tableId'] = 0 # Append the points // create a unique identifier that fits into an int64 id field vector_struct = { 'id': str(uuid4()), 'values': embedding, 'metadata': metadata, } cur_size = sys.getsizeof(vector_struct) sum_size += cur_size points_to_upsert.append(vector_struct) # Pinecone maxes out at 100 for bulk upsert but chose 50 to ensure it doesn't hit the limit if (len(points_to_upsert) >= 50) or (sum_size + cur_size > self.payload_limit): flush() cur_size = 0 sum_size = 0 flush() if len(objectIds): self.remove(list(objectIds.keys()))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_pinecone/pinecone.py` around lines 399 - 468, Update addChunks so flush only upserts pending points and does not call self.remove or clear objectIds between batches. After the loop’s final flush succeeds, delete the collected object IDs once, ensuring all new chunks are written before stale data is removed and failures leave existing data intact.nodes/src/nodes/store_pinecone/README.md (1)
52-59: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Profiles section default contradicts the Fields table and generated Schema.
This section states The default profile is
serverless-denseand marksserverless-denseasyes/default, but the Fields table (line 36) and the auto-generated Schema section (line 112) both say the actual default is"pod-based". Update the Profiles table/prose to match the real default so the docs don't mislead users about which deployment gets created whenprofileis unset.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_pinecone/README.md` around lines 52 - 59, Update the Profiles table and its concluding default-profile statement in the README to use pod-based as the default, matching the Fields table and generated Schema; mark serverless-dense as non-default and pod-based as default without changing their deployment descriptions.nodes/src/nodes/store_postgres/postgres.py (3)
123-127: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Add a connect timeout to the production DB connection.
IGlobal.validateConfig()explicitly usesconnect_timeout=3for its probe connection, but this constructor — the connection actually used for all reads/writes — has none. If the host is unreachable or the network stalls,psycopg2.connect()can hang indefinitely, blockingbeginGlobal()/pipeline startup on this thread.🔌 Proposed fix
self.client = psycopg2.connect( - dbname=self.database, user=self.user, password=self.password, host=self.host, port=self.port + dbname=self.database, user=self.user, password=self.password, host=self.host, port=self.port, + connect_timeout=10, )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.self.client = psycopg2.connect( dbname=self.database, user=self.user, password=self.password, host=self.host, port=self.port, connect_timeout=10, ) register_vector(self.client)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_postgres/postgres.py` around lines 123 - 127, Update the production connection in the store-postgres constructor to pass psycopg2’s connect_timeout parameter, matching the 3-second timeout used by IGlobal.validateConfig(). Keep the existing connection arguments and register_vector(self.client) behavior unchanged.
408-478: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
No rollback on failed statements — a single error poisons the shared connection for the rest of the Store's lifetime.
self.clientis a single persistent psycopg2 connection reused across_createCollection,addChunks,remove,markDeleted,markActive, and every search method. None of these ever callself.client.rollback(). If any statement inaddChunks's per-chunk insert loop (lines 424-439) raises (bad embedding size, unique/constraint violation, transient network error), Postgres puts the transaction into the aborted state; every subsequent query on this connection then fails with "current transaction is aborted" until the process is restarted — including unrelated reads for other objects. This is a real reliability risk for a long-lived store used across many pipeline instances.Wrap write paths in try/except and roll back on failure, e.g.:
try: ... self.client.commit() except Exception: self.client.rollback() raiseApply this consistently to
_createCollection,addChunks,remove,markDeleted, andmarkActive.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_postgres/postgres.py` around lines 408 - 478, Update the write methods _createCollection, addChunks, remove, markDeleted, and markActive to wrap their cursor operations and commit in try/except blocks; on any exception, call self.client.rollback() before re-raising the original exception. Preserve existing behavior on successful commits and early returns.
424-439: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift
Batch the chunk inserts instead of one
INSERTper chunk.The loop executes a separate round-trip
INSERTfor every chunk. For large ingests this is an unnecessary N-query hot path; considerpsycopg2.extras.execute_values(or building one multi-rowINSERT) to insert all chunks in a single statement.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_postgres/postgres.py` around lines 424 - 439, The chunk insertion flow using self.client.cursor and SQL_QUERIES['insert_chunk'] should batch all chunks into one database statement instead of executing one INSERT per iteration. Build the complete row values and shared column list first, then use psycopg2.extras.execute_values or an equivalent multi-row INSERT, preserving vectorSize and modelName defaults before committing once.nodes/test/conftest.py (1)
320-321: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Temporary skip — ensure it gets tracked for re-enablement.
store_elasticsearchis placed under the "exclude nodes with failing tests until they can be fixed" group, so its dynamically generated tests won't run in default CI. This matches the rename and the#1642dependency noted in the PR, but a lingering skip can mask regressions. Want me to open a follow-up issue to track re-enabling these tests once#1642lands?🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/test/conftest.py` around lines 320 - 321, Track the temporary exclusion of store_elasticsearch for re-enablement by adding a follow-up issue reference or TODO next to its entry in the failing-test exclusion group. Keep the exclusion unchanged until dependency `#1642` is resolved, and make the tracking explicit so the dynamically generated tests can be restored afterward.packages/docs/redirects.ts (1)
183-197: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Preserve Elasticsearch/OpenSearch variant deep links.
The provider-specific sources in Lines 187-195 currently redirect to the generic
/nodes/store_elasticsearchoverview, even though the documentation has nested Elasticsearch and OpenSearch pages. Keep/nodes/index_searchmapped to the overview, but route the provider-specific legacy paths to their corresponding nested pages.Proposed redirect split
{ "to": "/nodes/store_elasticsearch", "from": [ - "/nodes/index_search", - "/store/elasticsearch", - "/store/elasticsearch/elasticsearch" + "/nodes/index_search" + ] + }, + { + "to": "/nodes/store_elasticsearch/elasticsearch", + "from": [ + "/store/elasticsearch", + "/store/elasticsearch/elasticsearch" ] }, { - "to": "/nodes/store_elasticsearch", + "to": "/nodes/store_elasticsearch/opensearch", "from": [ "/store/opensearch", "/store/opensearch/opensearch"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.{ "to": "/nodes/store_elasticsearch", "from": [ "/nodes/index_search" ] }, { "to": "/nodes/store_elasticsearch/elasticsearch", "from": [ "/store/elasticsearch", "/store/elasticsearch/elasticsearch" ] }, { "to": "/nodes/store_elasticsearch/opensearch", "from": [ "/store/opensearch", "/store/opensearch/opensearch" ] },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/docs/redirects.ts` around lines 183 - 197, Update the redirect entries in the redirects configuration so /nodes/index_search continues targeting /nodes/store_elasticsearch, while Elasticsearch-specific legacy paths target the nested Elasticsearch page and OpenSearch-specific paths target the nested OpenSearch page. Preserve the existing source paths and split the provider-specific mappings accordingly.nodes/src/nodes/store_milvus/IGlobal.py (1)
74-77: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== file outline ==" ast-grep outline nodes/src/nodes/store_milvus/IGlobal.py --view expanded || true echo "== relevant IGlobal.py lines ==" sed -n '1,130p' nodes/src/nodes/store_milvus/IGlobal.py | cat -n echo "== locate milvus.py and runtime URI handling ==" fd -a 'milvus\.py$' . | sed "s#^\./##" | while read -r f; do echo "--- $f ---" rg -n "connections\.connect|uri\s*=|https://" "$f" -C 3 || true done echo "== python imports / dependencies ==" git ls-files | rg '(^|/)(pyproject\.toml|requirements.*\.txt|environment.*\.ya?ml|Pipfile)$' || true rg -n "pymilvus|milvus" -g 'pyproject.toml' -g 'requirements*.txt' -g '*.py' . | head -80 || trueRepository: rocketride-org/rocketride-server
Length of output: 22160
🌐 Web query:
pymilvus connections.connect Illegal uri expected form http[s]://💡 Result:
The error message "Illegal uri:..., expected form 'http[s]://[user:password@]example.com[:12345]'" in PyMilvus occurs because the uri parameter in connections.connect requires a properly formatted URI string [1][2]. To resolve this, ensure your uri string includes the protocol (http or https) [1][3]. Instead of passing just the host and port (e.g., "localhost:19530"), use the full URI format [1][4]: connections.connect( alias="default", uri="http://localhost:19530") Alternatively, you can provide the host and port as separate arguments, which does not require the URI string format [4][5]: connections.connect( alias="default", host="localhost", port="19530") If you are encountering this error while trying to connect to Milvus Lite (a local file-based instance), it may be because you are using an outdated version of PyMilvus [6][7]. Support for Milvus Lite was introduced in later versions; ensure your installation is at least version 2.4.2 or higher [6][7]: pip install pymilvus>=2.4.2
Citations:
- 1: milvus-io/milvus-docs#2420
- 2: milvus-io/milvus-lite#219
- 3: https://github.com/milvus-io/pymilvus/blob/c95290bd/pymilvus/orm/connections.py
- 4: https://milvus.io/api-reference/pymilvus/v2.6.x/ORM/Connections/connect.md
- 5: https://milvus.io/api-reference/pymilvus/v3.0.x/ORM/Connections/connect.md
- 6: https://milvus.io/docs/operational_faq.md
- 7: https://milvus.io/docs/v2.6.x/operational_faq.md
Use a scheme for the cloud validation URI.
connections.connect(uri=..., token=...)expectshttp[s]://host[:port]; a barehost:portraisesIllegal uri. The runtime Milvus client path addshttps://for cloud configs, so this save-time probe should use the same URI shape instead of failing valid token configurations.🐛 Proposed fix
if token: # Cloud: pass UI host and port verbatim via URI - uri = f'{host_raw}:{port}' + uri = f'https://{host_raw}:{port}' connections.connect(alias=alias, uri=uri, token=token, timeout=5)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.if token: # Cloud: pass UI host and port verbatim via URI uri = f'https://{host_raw}:{port}' connections.connect(alias=alias, uri=uri, token=token, timeout=5)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_milvus/IGlobal.py` around lines 74 - 77, Update the cloud validation branch in IGlobal.py to construct the URI with the same https:// scheme used by the runtime Milvus client before calling connections.connect. Preserve the raw host and port values while ensuring token-based cloud validation receives a valid http[s]://host[:port] URI.nodes/src/nodes/store_weaviate/IGlobal.py (1)
108-119: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Validation's cloud/local classification can diverge from the actual connection mode used at runtime.
validateConfiginfersis_local/is_cloudpurely from the host string (localhost/127.*), but the actualStore.__init__inweaviate.pydecides local vs. cloud fromconfig.get('mode')(profile == 'cloud'else local). Amode=localdeployment behind a non-loopback hostname (e.g. a Docker service name or private DNS entry) will be misclassified as cloud by this heuristic and probed via an HTTPS/v1/metarequest with Bearer auth — which doesn't match what the node will actually do at runtime — producing a confusing/incorrect validation warning for a legitimately-configured local instance.Prefer deriving
is_cloudfromconfig.get('mode') == 'cloud'(mirroringweaviate.py) instead of re-inferring it from the hostname.🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 113-113: Do not make http calls without encryption
Context: f'http://{host}'
Note: [CWE-319] Cleartext Transmission of Sensitive Information.(requests-http)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_weaviate/IGlobal.py` around lines 108 - 119, Update validateConfig’s cloud/local classification to derive is_cloud from config.get('mode') == 'cloud', matching Store.__init__ in weaviate.py, and derive is_local as its inverse. Remove the hostname-based localhost/127.* heuristic while preserving the existing URL construction for each resulting connection mode.nodes/src/nodes/store_weaviate/README.md (1)
126-129: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Generated "View source" link still points to the pre-rename
weaviatepath.Line 128 links to
.../tree/develop/nodes/src/nodes/weaviate, but this node now lives atnodes/src/nodes/store_weaviate. Since this sits inside the generated params block, it shouldn't be hand-edited — as per path instructions, "do not hand-edit content between<!-- ROCKETRIDE:GENERATED:PARAMS START -->and<!-- ROCKETRIDE:GENERATED:PARAMS END -->as it is regenerated bynodes:docs-generate." Re-running the generator against the renamed folder should pick up the correct path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_weaviate/README.md` around lines 126 - 129, Regenerate the documentation for the renamed store_weaviate node using the nodes:docs-generate workflow so the generated “View source” link targets nodes/src/nodes/store_weaviate instead of the obsolete nodes/src/nodes/weaviate path. Do not edit the content between the generated params markers manually.Source: Path instructions
nodes/src/nodes/store_weaviate/requirements.txt (1)
1-1: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== requirements file ==" if [ -f nodes/src/nodes/store_weaviate/requirements.txt ]; then cat -n nodes/src/nodes/store_weaviate/requirements.txt else fd -a 'requirements.txt' . | sed 's#^\./##' | head -50 fi echo echo "== authlib usages ==" rg -n "authlib|JWT|JWS|AuthCodeBackend|TokenAuth" nodes pyproject.toml requirements.txt 2>/dev/null || true echo echo "== pip index metadata if available ==" python3 - <<'PY' import urllib.request, json, re try: req = urllib.request.Request("https://pypi.org/pypi/authlib/json", headers={"User-Agent":"CodeRabbit verification"}) data = json.load(urllib.request.urlopen(req, timeout=20)) print("latest:", data["info"]["version"]) versions = sorted(data["releases"], key=lambda v: tuple(map(int_or_inf, re.split(r'[.\-_]', v)))) print("last 10 versions:", versions[-10:]) for ver in versions[-30:]: print(ver, "yanked:", bool(data["releases"][ver][0].get("yanked")) if data["releases"][ver] else None) except Exception as e: print("metadata fetch failed:", e) def int_or_inf(s): try: return int(s) except: return float('inf') PY echo echo "== pip index hashes for 1.3.1 and 1.7.2 ==" python3 - <<'PY' import urllib.request, json, re data = json.load(urllib.request.urlopen("https://pypi.org/pypi/authlib/json", timeout=20)) for ver in ("1.3.1", "1.7.2"): if ver in data.get("releases"): print(f"\n{ver}:") for rel in data["releases"][ver]: print(rel["packagetype"], rel["python_version"], rel["urls"] and rel["urls"][:1] if rel["urls"] else [], rel.get("yanked")) releases = data["releases"][ver] deps = [] seen=set() for src in releases: deps.append(src.get("requires_dist","")) print("requires_dist:", list(set(deps))) else: print(f"{ver}: missing") PYRepository: rocketride-org/rocketride-server
Length of output: 50389
Pin
authlibto 1.7.2 to avoid unresolved CVE-prone paths.The leaf requirement still allows older resolved versions, while
nodes/src/nodes/constraints.lockalready pinsauthlib==1.7.2, so update this file to use the same patched minimum.🛡️ Proposed fix
-authlib +authlib>=1.7.2📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.authlib>=1.7.2🧰 Tools
🪛 OSV Scanner (2.4.0)
[CRITICAL] 1-1: authlib 1.3.1: Authlib: JWS/JWT accepts unknown crit headers (RFC violation → possible authz bypass)
(PYSEC-2026-1200)
[CRITICAL] 1-1: authlib 1.3.1: Authlib has 1-click Account Takeover vulnerability
(PYSEC-2026-1201)
[CRITICAL] 1-1: authlib 1.3.1: Authlib : JWE zip=DEF decompression bomb enables DoS
(PYSEC-2026-1202)
[CRITICAL] 1-1: authlib 1.3.1: Authlib is vulnerable to Denial of Service via Oversized JOSE Segments
(PYSEC-2026-1203)
[CRITICAL] 1-1: authlib 1.3.1: undefined
(PYSEC-2026-188)
[CRITICAL] 1-1: authlib 1.3.1: undefined
(PYSEC-2026-2116)
[CRITICAL] 1-1: authlib 1.3.1: undefined
(PYSEC-2026-2117)
[CRITICAL] 1-1: authlib 1.3.1: undefined
(PYSEC-2026-2119)
[CRITICAL] 1-1: authlib 1.3.1: undefined
(PYSEC-2026-25)
[CRITICAL] 1-1: authlib 1.3.1: Authlib JWS JWK Header Injection: Signature Verification Bypass
(PYSEC-2026-287)
[CRITICAL] 1-1: authlib 1.3.1: Authlib Vulnerable to JWE RSA1_5 Bleichenbacher Padding Oracle
(GHSA-7432-952r-cw78)
[CRITICAL] 1-1: authlib 1.3.1: Authlib: JWS/JWT accepts unknown crit headers (RFC violation → possible authz bypass)
(GHSA-9ggr-2464-2j32)
[CRITICAL] 1-1: authlib 1.3.1: Authlib has 1-click Account Takeover vulnerability
(GHSA-fg6f-75jq-6523)
[CRITICAL] 1-1: authlib 1.3.1: Authlib : JWE zip=DEF decompression bomb enables DoS
(GHSA-g7f3-828f-7h7m)
[CRITICAL] 1-1: authlib 1.3.1: Authlib: Cross-site request forging when using cache
(GHSA-jj8c-mmj3-mmgv)
[CRITICAL] 1-1: authlib 1.3.1: Authlib: Fail-Open Cryptographic Verification in OIDC Hash Binding
(GHSA-m344-f55w-2m6j)
[CRITICAL] 1-1: authlib 1.3.1: Authlib is vulnerable to Denial of Service via Oversized JOSE Segments
(GHSA-pq5p-34cr-23v9)
[CRITICAL] 1-1: authlib 1.3.1: Authlib OIDC Implicit/Hybrid Authorization Vulnerable to Open Redirect
(GHSA-r95x-qfjj-fjj2)
[CRITICAL] 1-1: authlib 1.3.1: Authlib OAuth 2.0 has Open Redirect in Authorization API that allows attacker-controlled redirect_uri through unsupported response_type
(GHSA-w8p2-r796-3vmq)
[CRITICAL] 1-1: authlib 1.3.1: Authlib JWS JWK Header Injection: Signature Verification Bypass
(GHSA-wvwj-cvrp-7pv5)
[CRITICAL] 1-1: h11 0.8.1: h11 accepts some malformed Chunked-Encoding bodies
(PYSEC-2026-348)
[CRITICAL] 1-1: h11 0.8.1: h11 accepts some malformed Chunked-Encoding bodies
(GHSA-vqfr-h8mv-ghfj)
[MEDIUM] 1-1: h2 3.2.0: h2 allows HTTP Request Smuggling due to illegal characters in headers
(PYSEC-2026-1435)
[MEDIUM] 1-1: h2 3.2.0: h2 allows HTTP Request Smuggling due to illegal characters in headers
(GHSA-847f-9342-265h)
[HIGH] 1-1: idna 2.9.0: undefined
(PYSEC-2024-60)
[HIGH] 1-1: idna 2.9.0: undefined
(PYSEC-2026-215)
[HIGH] 1-1: idna 2.9.0: Internationalized Domain Names in Applications (IDNA): Specially crafted inputs to idna.encode() can bypass CVE-2024-3651 fix
(GHSA-65pc-fj4g-8rjx)
[HIGH] 1-1: idna 2.9.0: Internationalized Domain Names in Applications (IDNA) vulnerable to denial of service from specially crafted inputs to idna.encode
(GHSA-jjg7-2v4v-x38h)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_weaviate/requirements.txt` at line 1, Update the authlib dependency declaration in requirements.txt to pin it to version 1.7.2, matching the existing authlib==1.7.2 constraint in constraints.lock.nodes/src/nodes/store_weaviate/weaviate.py (5)
267-277: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail echo "== file excerpt ==" sed -n '230,295p' nodes/src/nodes/store_weaviate/weaviate.py | cat -n echo "== search similarity/default/documentstore threshold ==" rg -n "similarity|threshold_search|distance|certainty|cosine" nodes src -S || true echo "== python files outline around store weaviate ==" ast-grep outline nodes/src/nodes/store_weaviate/weaviate.py --view expanded || true echo "== git status/stat ==" git diff --stat || true git diff -- nodes/src/nodes/store_weaviate/weaviate.py | sed -n '1,220p' || true echo "== behavioral probe for formulas ==" python3 - <<'PY' import math, np vals = [0, 0.5, 1, 1.5, 2] for d in vals: current = (d + 1) / 2 proposed = 1 - (d / 2) sigmoid = float(1.0 / (1.0 + math.exp(d / -100))) print(f"distance={d:.2f} current={current:.2f} proposed={proposed:.2f} sigmoid={sigmoid:.6f}") PYRepository: rocketride-org/rocketride-server
Length of output: 47150
🌐 Web query:
Weaviate cosine distance range 0 to 2 certainty formula 1 - distance / 2💡 Result:
In Weaviate, when the cosine distance metric is used, the distance value ranges from 0 to 2 [1][2]. The certainty score is a normalized value between 0 and 1 that represents similarity, where 1 indicates identical vectors and 0 indicates opposite vectors [3][2]. The formula to convert a cosine distance to a certainty score is: certainty = 1 - (distance / 2) [4] This conversion is specific to the cosine distance metric [3][2]. Because not all distance metrics in Weaviate have a fixed range, the use of distance is generally preferred over certainty in modern Weaviate versions, though certainty remains supported for backward compatibility when using cosine distance [4][2].
Citations:
- 1: https://docs.weaviate.io/weaviate/config-refs/distances
- 2: https://archive.docs.weaviate.io/weaviate/config-refs/distances
- 3: https://docs.weaviate.io/weaviate/api/graphql/additional-properties
- 4: https://forum.weaviate.io/t/distance-vs-certainty-scores/258
🏁 Script executed:
#!/bin/bash set -euo pipefail python3 - <<'PY' import math def current(d): return (d + 1) / 2 def proposed(d): return 1 - (d / 2) def sigmoid(d): return 1.0 / (1.0 + math.exp(d / -100)) for d in [0, 0.5, 1.0, 1.99, 2.0, 4.0]: print(f"distance={d:.2f} current={current(d):.2f} proposed={proposed(d):.2f} sigmoid={sigmoid(d):.6f}") PY echo "== deterministic property check ==" python3 - <<'PY' def current(d): return (d + 1) / 2 def proposed(d): return 1 - (d / 2) distances = [0, 1e-6, 0.20, 0.5, 1.0, 1.80, 1.999, 2.0] for d in distances: print({ 'distance': d, 'current_identity': current(d), 'proposed_identity': proposed(d), 'current_in_0_1': 0 <= current(d) <= 1, 'proposed_in_0_1': 0 <= proposed(d) <= 1, 'current_higher_for_d_greater_1': current(d) >= 1 for d in distances }) PYRepository: rocketride-org/rocketride-server
Length of output: 737
Normalize Weaviate cosine distances with certainty semantics.
Weaviate cosine distances range from
0to2, so the current(distance + 1) / 2mapping makes identical vectors score0.5, passes dissimilar vectors like distance1.8as1.4, and produces scores outside the[0, 1]threshold space. Use Weaviate’s cosine certainty formula:1 - distance / 2.🐛 Proposed fix
if point.metadata.distance is not None: if self.similarity == 'cosine': - score = (point.metadata.distance + 1) / 2 + score = 1 - (point.metadata.distance / 2) else:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.if point.metadata.distance is not None: if self.similarity == 'cosine': score = 1 - (point.metadata.distance / 2) else: score = float(1.0 / (1.0 + np.exp(point.metadata.distance / -100))) # Ignore it if it doesn't have a high enough score if score < 0.20: continue else: score = 0.0🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_weaviate/weaviate.py` around lines 267 - 277, Update the cosine branch in the score calculation around self.similarity to normalize Weaviate distances using certainty semantics: compute the score as 1 minus distance divided by 2. Keep the existing non-cosine calculation, threshold check, and missing-distance behavior unchanged.
302-326: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash # Confirm QuestionText's __str__ behavior and how sibling stores build keyword filters from it. rg -n -A5 -B2 'class QuestionText' --type=py rg -n -A3 -B3 "searchKeyword" packages/ai/src nodes/src/nodes --type=py -g '!store_weaviate'Repository: rocketride-org/rocketride-server
Length of output: 170
🏁 Script executed:
#!/bin/bash set -e echo "== files ==" git ls-files | rg '(^|/)weaviate\.py$|QuestionText|IGlobal|IGlobal|keyword' echo "== target file outline/size ==" find . -path '*/store_weaviate/weaviate.py' -type f -print | while read f; do echo "FILE: $f" wc -l "$f" ast-grep outline "$f" --match searchKeyword --view expanded || true done echo "== QuestionText definitions/usages ==" rg -n -A10 -B5 'QuestionText' . --glob '*.py' || true echo "== searchKeyword usages ==" rg -n -A8 -B6 'searchKeyword' . --glob '*.py' || true echo "== keyword filter patterns in Python files ==" rg -n -A4 -B4 "like\\(f'\\*\\{|like\\(\\*|by_property\\('content'\\)" . --glob '*.py' || true echo "== relevant target lines ==" for f in $(git ls-files | rg '(^|/)weaviate\.py$'); do echo "FILE: $f" sed -n '280,340p' "$f" doneRepository: rocketride-org/rocketride-server
Length of output: 50388
🏁 Script executed:
#!/bin/bash set -e echo "== QuestionText definition ==" sed -n '262,295p' packages/client-python/src/rocketride/schema/question.py echo "== store_weaviate relevant lines ==" sed -n '288,330p' nodes/src/nodes/store_weaviate/weaviate.py echo "== sibling keyword filter implementations concise ==" python3 - <<'PY' from pathlib import Path paths = [ 'nodes/src/nodes/store_atlas/atlas.py', 'nodes/src/nodes/store_chroma/chroma.py', 'nodes/src/nodes/store_elasticsearch/elasticsearch_store.py', 'nodes/src/nodes/store_milvus/milvus.py', 'nodes/src/nodes/store_pinecone/pinecone.py', 'nodes/src/nodes/store_postgres/postgres.py', 'nodes/src/nodes/store_weaviate/weaviate.py', ] text = "class DocumentStoreBase\n@abstractmethod\n def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:" for p in paths: s = Path(p).read_text() idx = s.find(text) if idx == -1: print(f"{p}: NOT FOUND") continue body = s[idx:s.find('\n def searchSemantic', idx)] query_refs = sorted(set(l.strip() for l in body.splitlines() if 'query' in l or query := getattr(l,l, None))) print(f"\n--- {p} ---") print(body.strip()) PY echo "== pydantic BaseModel repr/str probe if installed ==" python3 - <<'PY' try: from pydantic import BaseModel except Exception as e: print(f"pydantic import failed: {type(e).__name__}: {e}") raise SystemExit(0) class QuestionText(BaseModel): text: str embedding_model: str = 'default' embedding: list | None = None instance = QuestionText(text='hello') print("Instance:", instance) print("str:", str(instance)) print("repr:", repr(instance)) PYRepository: rocketride-org/rocketride-server
Length of output: 3071
🏁 Script executed:
#!/bin/bash set -e echo "== concise sibling keyword implementations ==" python3 - <<'PY' from pathlib import Path marker = "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:" markers = [ ("nodes/src/nodes/store_atlas/atlas.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n \"\"\"Perform keyword search using MongoDB text search.\"\"\""), ("nodes/src/nodes/store_chroma/chroma.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n \"\"\"\n Keyword search.\n \"\"\""), ("nodes/src/nodes/store_elasticsearch/elasticsearch_store.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n \"\"\"\n Keyword search using Elasticsearch text search.\n \"\"\""), ("nodes/src/nodes/store_milvus/milvus.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n \"\"\"\n Keyword search.\n \"\"\""), ("nodes/src/nodes/store_pinecone/pinecone.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n \"\"\"\n Keyword search.\n \"\"\""), ("nodes/src/nodes/store_postgres/postgres.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n \"\"\"\n Perform a keyword search."), ("nodes/src/nodes/store_weaviate/weaviate.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n \"\"\"\n Keyword search.\n \"\"\""), ] for path, start in markers: s = Path(path).read_text() idx = s.find(start) if idx == -1: print(f"{path}: NOT FOUND") continue end = s.find("\n def searchSemantic", idx) print(f"\n--- {path} ---") print(s[idx:end].strip()) PY echo "== pydantic BaseModel repr/str probe if installed ==" python3 - <<'PY' try: from pydantic import BaseModel except Exception as e: print(f"pydantic import failed: {type(e).__name__}: {e}") raise SystemExit(0) class QuestionText(BaseModel): text: str instance = QuestionText(text='hello world') print("Instance:", instance) print("str:", str(instance)) print("repr:", repr(instance)) PY echo "== pydantic install-only source facts if already installed ==" python3 - <<'PY' import inspect try: from pydantic import BaseModel except Exception as e: print("not installed/unavailable") raise SystemExit(0) cls = BaseModel print("BaseModel MRO:", [c.__name__ for c in cls.mro()]) print("has __repr__:", hasattr(cls, '__repr__')) try: print("__repr__ from:", inspect.getfile(cls.__repr__)) except Exception as e: print("__repr__ source unavailable:", e) PYRepository: rocketride-org/rocketride-server
Length of output: 13079
🌐 Web query:
Pydantic v2 BaseModel __str__ __repr__ default behavior ModelStr💡 Result:
In Pydantic v2, BaseModel instances provide default implementations for str and repr to provide a human-readable representation of the model instance and its fields [1]. Default Behavior: - The repr method returns a string representation of the model, typically in the format ModelName(field1=value1, field2=value2,...) [1]. - The str method by default delegates to the repr method, meaning printing an instance of a Pydantic model will display the same information as its representation [2]. Customization: - Field Representation: You can control which fields appear in the output of str and repr using the Field(repr=...) argument [3]. By setting repr=False in a Field definition, that field will be excluded from the generated string/representation [3]. - Overriding: You can override str or repr directly in your BaseModel subclass if you need custom logic for how your model is displayed or printed [2]. Regarding ModelStr: - Pydantic does not have a native type called ModelStr. It is likely you are referring to the standard str type used as a type hint for fields, or potentially a misunderstanding of how string fields are handled [4][5]. Pydantic v2 handles standard Python strings as-is, while providing specialized string types like EmailStr or ImportString for specific validation needs [5]. If you were referring to an issue where custom subclassed string types appear to lose their specific class type information after model construction, this is a known aspect of Pydantic's design regarding validation and coercion [6].
Citations:
- 1: pydantic/pydantic#884
- 2: pydantic/pydantic#4133
- 3: https://pydantic.dev/docs/validation/latest/concepts/fields/
- 4: https://pydantic.dev/docs/validation/2.4/concepts/models/
- 5: https://pydantic.dev/docs/validation/2.2/usage/types/string_types/
- 6: pydantic/pydantic#6702
Use
query.textin the Weaviate keyword filter.
QuestionTextis a Pydantic model, sof'*{query}*'produces a model repr such asQuestionText(text=..., embedding_model=...)rather than the search text. Change the filter toFilter.by_property('content').like(f'*{query.text}*')to match the question text directly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_weaviate/weaviate.py` around lines 302 - 326, The searchKeyword method currently interpolates the entire QuestionText model into the Weaviate content filter. Update the filter construction to use query.text so the wildcard pattern matches the question text directly.
415-469: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Delete-then-insert in
addChunksis not atomic; a partial batch failure can lose data.Existing chunks for the incoming
objectIdsare deleted (line 442) before the new batch is inserted (445-463). If the batch import partially fails, the exception raised at line 467 surfaces only after the old chunks are already gone and some new chunks may be missing — leaving the document's chunk set corrupted with no rollback. The existing# TODO: Start discussion about better use of upsert()(line 436) hints at this being a known gap, though framed around performance rather than atomicity/data-loss risk.Happy to help prototype a safer approach (e.g., insert-then-delete-old, or use Weaviate's UUID-based upsert so re-ingest overwrites in place without a delete step) if useful — want me to open an issue to track it?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_weaviate/weaviate.py` around lines 415 - 469, Update addChunks to avoid deleting existing chunks before the replacement batch has been imported successfully. Use an atomic or rollback-safe replacement strategy—prefer UUID-based upsert when available, otherwise insert the new chunks first and delete prior chunks only after the batch completes without failed_objects—while preserving the existing objectId replacement semantics.
489-514: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win
markDeleted/markActivesilently truncate to 10 chunks — missinglimitonfetch_objects.Both methods call
self.collectionObj.query.fetch_objects(filters=Filter.any_of(objectIdsFilter))without alimit. The Weaviate v4 Python client defaultsfetch_objects()to 10 results when no limit is given — confirmed on the community forum: "I have a simple collection with 1700 items... print(len(all_items)) -> this prints 10" unlesslimitis set. For any document with more than 10 chunks, only the first 10 will haveisDeletedtoggled, leaving the rest in the wrong soft-delete state — a silent, hard-to-detect data-integrity bug. Compare withrender()in this same file, which already paginates in windows for exactly this reason.🐛 Proposed fix: paginate until exhausted
- objectIdsFilter: List[Filter] = [] - for objectId in objectIds: - objectIdsFilter.append(Filter.by_property('objectId').equal(objectId)) - points = self.collectionObj.query.fetch_objects(filters=Filter.any_of(objectIdsFilter)) - for point in points.objects: - self.collectionObj.data.update( - uuid=point.uuid, - properties={ - 'isDeleted': True, - }, - ) - return + objectIdsFilter: List[Filter] = [] + for objectId in objectIds: + objectIdsFilter.append(Filter.by_property('objectId').equal(objectId)) + combined_filter = Filter.any_of(objectIdsFilter) + while True: + points = self.collectionObj.query.fetch_objects(filters=combined_filter, limit=10000) + if not points.objects: + break + for point in points.objects: + self.collectionObj.data.update(uuid=point.uuid, properties={'isDeleted': True}) + if len(points.objects) < 10000: + break + return(apply the analogous change to
markActive)Also applies to: 516-541
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_weaviate/weaviate.py` around lines 489 - 514, Update markDeleted and markActive so their fetch_objects queries do not rely on the default 10-result limit. Paginate through matching objects in windows until no results remain, applying the existing isDeleted update to every returned object. Preserve the collection-existence and empty-objectIds early returns.
543-606: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
render()preallocates a multi-million-element list on every window iteration.
text = [''] * self.renderChunkSizeat line 573 allocates a Python list sized byrenderChunkSize, whose default is32 * 1024 * 1024(~33.5M entries). Every call torender()— even for a document with a handful of chunks — allocates this ~33M-element list at least once (and repeats it per pagination window), burning significant CPU/memory on every render request. The naming (renderChunkSize) and magnitude (32*1024*1024, looking like a byte-size constant) suggest the "window" was never intended to hold literally that many chunks.Consider keying by
chunk - offsetinto adictinstead of preallocating a fixed-size list, so memory scales with the actual number of returned points rather than the configured window size.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/store_weaviate/weaviate.py` around lines 543 - 606, Update render() to replace the fixed-size text list allocation with a sparse dictionary keyed by each point’s chunk - offset, so memory usage scales with returned chunks rather than renderChunkSize. Preserve chunk ordering, missing-chunk handling, lastIndex/numberOfItems calculation, joined output, pagination, and callback behavior.
asclearuc
left a comment
There was a problem hiding this comment.
Thanks @dsapandora for the careful staging of this rename — the compat story holds up. I verified the core claim: node identity for saved pipes is the per-services.json protocol/title (proof: store_elasticsearch registers two providers from one folder), protocols/prefixes are untouched, and examples/db_arango.pipe still resolves. Redirects are complete and CI is fully green.
Requesting changes on the documentation half of the rename, per the co-located-docs rule (same change, not follow-up):
- 9 node READMEs keep the old identity — every renamed node except
store_postgresstill has the old name as its title (# milvus,# astra_db,# db_arango, …) and a "View source" link to the old folder path, which 404s once this merges. Please apply the same fix you did forstore_postgres. docs/README-nodes.mdcatalog (lines 9, 137–145) still lists all nine old node IDs. On the Copilot thread you wrote it regenerates fromservices.json— I checked the tree and found no generator;docs.ymlonly runsbuilder docs:build, which doesn't touch this file. Copilot's comment stands; the table needs a manual update.tools/dependabot-smoke/smoke-nodes.sh:31still keysnodes/src/nodes/index_search— that directory won't exist after merge, so the opensearchpy smoke check silently dies. Should bestore_elasticsearch.- PR description: still says
vectordb_prefix (actual:store_), says test folders follow the new names (they intentionally don't, per your own reply on the Copilot thread), and counts eight stores (there are nine). Please refresh it — it's what release notes will show.
Non-blocking: a few comments made stale by the rename (graph_falkordb/services.json:203, graph_hydradb/services.json:16-17, packages/ai/.../graph/__init__.py:30) and content-static links to old /nodes/* URLs (redirects cover them).
Co-located-docs follow-through for the rename in this PR (per asclearuc review): - 10 node READMEs still carried the old title and a "View source" link to the old folder (# milvus -> # store_milvus, .../nodes/milvus -> .../nodes/store_milvus), which 404s after merge. store_postgres was already done. - docs/README-nodes.md: the Vector Databases & Stores table listed the old folder-derived ids; refreshed to store_*. The catalog is hand-maintained (no generator), so the misleading "generated" note was corrected to match. - tools/dependabot-smoke/smoke-nodes.sh: index_search -> store_elasticsearch so the opensearchpy smoke check keeps resolving after the folder rename.
Now that #1642's collection-time guard undoes the sys.modules stub leaks, the reordered pytest collection is harmless, so the test folders can follow their nodes: nodes/test/{astra_db,atlas,chroma,milvus,pinecone,qdrant,weaviate} -> store_*. Tests load src via ../../src/nodes/store_* (unchanged depth), so only the two ocr breadcrumbs and two stale "Usage" docstrings needed the new paths.
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
nodes/src/nodes/graph_hydradb/README.md (2)
66-67: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winRegenerate the missing configuration schema.
The generated-parameters block is empty, so published documentation omits the node’s API key, database, collection, and max-results schema. Run
nodes:docs-generateand commit the generated content; do not hand-edit between these markers.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/graph_hydradb/README.md` around lines 66 - 67, Regenerate the missing configuration schema in the ROCKETRIDE:GENERATED:PARAMS block by running nodes:docs-generate, then commit the generated API key, database, collection, and max-results documentation without hand-editing content between the markers.Source: Coding guidelines
27-27: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winDo not promise memory zeroization without implementing it.
endGlobal()only assignsself.client.api_key = ''and drops the client reference, which does not guarantee all existing secret copies are removed. Either implement actual secret zeroization or change the README/IGlobal.pywording to say the client/key reference is cleared/dropped.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/src/nodes/graph_hydradb/README.md` at line 27, Update the API-key lifecycle wording in README.md and the corresponding IGlobal.py documentation to state only that the client/key reference is cleared or dropped during teardown, unless endGlobal() is changed to implement actual zeroization of every secret copy. Remove the unsupported promise that the key is cleared from memory.nodes/test/store_milvus/test_filter_injection.py (1)
93-97: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winAssert escaping rather than remove the payload under test.
Line 97 removes
'|| 1==1'before checking for it, so the assertion always passes regardless of sanitization.Proposed fix
- self.assertNotIn('|| 1==1', filter_expr.replace(_escape_milvus_str('|| 1==1'), '')) + self.assertEqual(filter_expr, "content like '%%\\' || 1==1 || content like \\'%%'")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@nodes/test/store_milvus/test_filter_injection.py` around lines 93 - 97, Update test_inject_via_keyword_search so it asserts the constructed filter_expr directly, without replacing or removing the payload fragment under test. Verify that the escaped result does not contain the unescaped injection sequence while preserving the existing payload and _escape_milvus_str setup.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@nodes/test/store_milvus/test_addchunks_return.py`:
- Line 19: Enable the regression test by removing the pytest skip marker and
refactoring its imports or fixtures as needed so it runs successfully. Preserve
the test’s coverage of the addChunks() return contract.
In `@nodes/test/store_milvus/test_filter_injection.py`:
- Line 14: Scope the temporary sys.path insertion around the milvus provider
import and restore the original import path immediately afterward in
nodes/test/store_milvus/test_filter_injection.py lines 14-14. Apply the same
scoped cleanup around the provider import in
nodes/test/store_milvus/test_search_semantic.py lines 13-13, ensuring test
collection does not retain either path modification.
---
Outside diff comments:
In `@nodes/src/nodes/graph_hydradb/README.md`:
- Around line 66-67: Regenerate the missing configuration schema in the
ROCKETRIDE:GENERATED:PARAMS block by running nodes:docs-generate, then commit
the generated API key, database, collection, and max-results documentation
without hand-editing content between the markers.
- Line 27: Update the API-key lifecycle wording in README.md and the
corresponding IGlobal.py documentation to state only that the client/key
reference is cleared or dropped during teardown, unless endGlobal() is changed
to implement actual zeroization of every secret copy. Remove the unsupported
promise that the key is cleared from memory.
In `@nodes/test/store_milvus/test_filter_injection.py`:
- Around line 93-97: Update test_inject_via_keyword_search so it asserts the
constructed filter_expr directly, without replacing or removing the payload
fragment under test. Verify that the escaped result does not contain the
unescaped injection sequence while preserving the existing payload and
_escape_milvus_str setup.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 71357b06-1e9b-4ef8-a2a9-33055b5a488b
📒 Files selected for processing (31)
docs/README-nodes.mdnodes/src/nodes/graph_arango/README.mdnodes/src/nodes/graph_hydradb/README.mdnodes/src/nodes/store_astra/README.mdnodes/src/nodes/store_atlas/README.mdnodes/src/nodes/store_chroma/README.mdnodes/src/nodes/store_elasticsearch/README.mdnodes/src/nodes/store_milvus/README.mdnodes/src/nodes/store_pinecone/README.mdnodes/src/nodes/store_qdrant/README.mdnodes/src/nodes/store_weaviate/README.mdnodes/test/ocr/test_model_server_ocr.pynodes/test/store_astra/__init__.pynodes/test/store_astra/test_convert_filter.pynodes/test/store_atlas/__init__.pynodes/test/store_atlas/test_isdeleted_filter.pynodes/test/store_chroma/__init__.pynodes/test/store_chroma/test_coerce_port.pynodes/test/store_chroma/test_convert_filter_explicit_none.pynodes/test/store_milvus/__init__.pynodes/test/store_milvus/test_addchunks_return.pynodes/test/store_milvus/test_filter_injection.pynodes/test/store_milvus/test_search_semantic.pynodes/test/store_pinecone/__init__.pynodes/test/store_pinecone/test_record_updates.pynodes/test/store_qdrant/__init__.pynodes/test/store_qdrant/test_mark_deleted.pynodes/test/store_qdrant/test_search_threshold.pynodes/test/store_weaviate/__init__.pynodes/test/store_weaviate/test_record_updates.pytools/dependabot-smoke/smoke-nodes.sh
|
Thanks Alexandru — all fixed on READMEs (titles + View-source links), the Also went one further: now that #1642's guard kills the Left the non-blocking stale comments out to keep this renames-only; happy to fold them in if you'd rather. |
asclearuc
left a comment
There was a problem hiding this comment.
Thank you @dsapandora
Approving
…s/ variant dir (#1757) * fix(docs): unblock docs deploys broken since #1570 docs:gather's package-mount pass globs {nodes,packages,apps}/**/docs/** and throws on any match without a declared mount. tool_google_workspace's Google Docs variant lives at nodes/src/nodes/tool_google_workspace/docs/, so its README matches the glob and every docs build since 2026-07-17 has aborted, leaving docs.rocketride.org stale since the last successful deploy on 07-15. Exclude nodes/src/nodes/** and nodes/test/** from the package-mount pass; markdown under node sources is staged by the dedicated nodes contributor. Also fix the stale internal links that the revived build surfaces: four spine pages still linked pre-rename routes from #1636/#1611 (/nodes/qdrant, /nodes/pinecone, /nodes/milvus, /nodes/db_neo4j), which now fail the broken-links check. Point them at the store_*/graph_* routes and update the neo4j page's node name. Verified locally: builder docs:build compiles clean; gather stages 170 pages including all 118 node docs. Fixes #1756 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(build): correct the IGNORE comment about who stages node markdown Review pointed out the comment overclaimed: variant READMEs are only staged when the node's top-level README carries the generated markers, so tool_google_workspace's variant docs are currently staged by nobody. Say what the contributor pass actually does instead of implying full coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…o graph_* (rocketride-org#1636) * refactor(nodes): rename vector stores to store_* and graph-first db to graph_* Rename the nine vector-store nodes to the store_ prefix (matching their `store` classType) and the two graph-first databases (ArangoDB, HydraDB) to graph_. Update each services.json path to the new python module and add graph to the ArangoDB/HydraDB classType (database and tool kept). protocol and prefix are left untouched, so saved pipes keep resolving; no services-version bump. Docs redirects map the old /nodes/<old> URLs to the new ones. Test module imports and path references follow the renamed src folders. constraints.lock and docs static pages are regenerated by CI. Node test folders keep their original names so pytest collection order is unchanged. * docs(nodes): align renamed node docs with store_*/graph_* identity Co-located-docs follow-through for the rename in this PR (per asclearuc review): - 10 node READMEs still carried the old title and a "View source" link to the old folder (# milvus -> # store_milvus, .../nodes/milvus -> .../nodes/store_milvus), which 404s after merge. store_postgres was already done. - docs/README-nodes.md: the Vector Databases & Stores table listed the old folder-derived ids; refreshed to store_*. The catalog is hand-maintained (no generator), so the misleading "generated" note was corrected to match. - tools/dependabot-smoke/smoke-nodes.sh: index_search -> store_elasticsearch so the opensearchpy smoke check keeps resolving after the folder rename. * test(nodes): rename test folders to store_* to match the renamed nodes Now that rocketride-org#1642's collection-time guard undoes the sys.modules stub leaks, the reordered pytest collection is harmless, so the test folders can follow their nodes: nodes/test/{astra_db,atlas,chroma,milvus,pinecone,qdrant,weaviate} -> store_*. Tests load src via ../../src/nodes/store_* (unchanged depth), so only the two ocr breadcrumbs and two stale "Usage" docstrings needed the new paths.
…s/ variant dir (rocketride-org#1757) * fix(docs): unblock docs deploys broken since rocketride-org#1570 docs:gather's package-mount pass globs {nodes,packages,apps}/**/docs/** and throws on any match without a declared mount. tool_google_workspace's Google Docs variant lives at nodes/src/nodes/tool_google_workspace/docs/, so its README matches the glob and every docs build since 2026-07-17 has aborted, leaving docs.rocketride.org stale since the last successful deploy on 07-15. Exclude nodes/src/nodes/** and nodes/test/** from the package-mount pass; markdown under node sources is staged by the dedicated nodes contributor. Also fix the stale internal links that the revived build surfaces: four spine pages still linked pre-rename routes from rocketride-org#1636/rocketride-org#1611 (/nodes/qdrant, /nodes/pinecone, /nodes/milvus, /nodes/db_neo4j), which now fail the broken-links check. Point them at the store_*/graph_* routes and update the neo4j page's node name. Verified locally: builder docs:build compiles clean; gather stages 170 pages including all 118 node docs. Fixes rocketride-org#1756 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(build): correct the IGNORE comment about who stages node markdown Review pointed out the comment overclaimed: variant READMEs are only staged when the node's top-level README carries the generated markers, so tool_google_workspace's variant docs are currently staged by nobody. Say what the contributor pass actually does instead of implying full coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Summary
store_prefix (matching thestoreclassType) and the two graph-first databases (ArangoDB, HydraDB) tograph_, following the establisheddb_/graph_family convention; addgraphto the ArangoDB/HydraDBclassType(keptdatabaseandtool).path(python module) andclassTypechange —protocolandprefixare left untouched, so saved.pipeconfigs keep resolving and no services-version bump is needed.store_*to match their nodes (7 folders), safe now that fix(nodes, test): scope core-module stubs so they don't leak across tests #1642's collection-time guard undoes thesys.modulesleaks a reordered collection would surface. Tests load src via../../src/nodes/store_*(unchanged depth);redirects.tsmaps the old/nodes/<old>doc URLs to the new ones.README-nodes.mdcatalog (hand-maintained, no generator), and the dependabot smoke map.constraints.lockis regenerated by CI (lock-node-deps.yml).This is PR 1 of the staged plan (renames only, non-breaking). Extracting a
storebase, migrating the vector nodes onto it, and the provider renames land in later PRs.Type
refactor (node naming consistency)
Testing
./builder testpasses (runs in CI on this PR)Checklist
protocol/prefixunchanged, saved pipes keep resolving; only node/catalog names changeLinked Discussion
Design and rationale: #1635 (node naming /
classTypeunification). Not linked to an issue — this PR implements the first stage proposed in that discussion.Summary by CodeRabbit