Skip to content

refactor(nodes): rename vector stores to store_* and graph-first db to graph_* - #1636

Merged
dsapandora merged 3 commits into
developfrom
refactor/node-family-renames
Jul 23, 2026
Merged

dsapandora merged 3 commits into
developfrom
refactor/node-family-renames

Conversation

@dsapandora

@dsapandora dsapandora commented Jul 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Rename the nine vector-store nodes to the store_ prefix (matching the store classType) and the two graph-first databases (ArangoDB, HydraDB) to graph_, following the established db_/graph_ family convention; add graph to the ArangoDB/HydraDB classType (kept database and tool).
  • Only the folder name, path (python module) and classType change — protocol and prefix are left untouched, so saved .pipe configs keep resolving and no services-version bump is needed.
  • Test folders are renamed to store_* to match their nodes (7 folders), safe now that fix(nodes, test): scope core-module stubs so they don't leak across tests #1642's collection-time guard undoes the sys.modules leaks a reordered collection would surface. Tests load src via ../../src/nodes/store_* (unchanged depth); redirects.ts maps the old /nodes/<old> doc URLs to the new ones.
  • Co-located docs updated in this same PR: each node README title + "View source" link, the README-nodes.md catalog (hand-maintained, no generator), and the dependabot smoke map. constraints.lock is regenerated by CI (lock-node-deps.yml).

This is PR 1 of the staged plan (renames only, non-breaking). Extracting a store base, migrating the vector nodes onto it, and the provider renames land in later PRs.

Type

refactor (node naming consistency)

Testing

  • Tests added or updated (test folders/files renamed, path refs + skip-list updated)
  • Tested locally (lint hooks green: gitleaks / ruff-check / ruff-format; full suite runs in CI)
  • ./builder test passes (runs in CI on this PR)

Checklist

  • Commit messages follow conventional commits
  • No secrets or credentials included
  • Wiki updated (if applicable) — N/A
  • Breaking changes documented — no runtime break: protocol/prefix unchanged, saved pipes keep resolving; only node/catalog names change

Linked Discussion

Design and rationale: #1635 (node naming / classType unification). Not linked to an issue — this PR implements the first stage proposed in that discussion.

Summary by CodeRabbit

  • New Features
    • Added graph database integrations for ArangoDB and HydraDB, including schema discovery, natural-language queries, memory storage, graph queries, and controlled execution.
    • Added unified document and vector store nodes for Astra, Atlas, Chroma, Elasticsearch/OpenSearch, Milvus, Pinecone, PostgreSQL, Qdrant, and Weaviate.
    • Added text and semantic search, document ingestion, retrieval, deletion, and rendering capabilities across supported stores.
  • Documentation
    • Updated node names, source links, catalog entries, and legacy redirects.

@github-actions github-actions Bot added docs Documentation module:nodes Python pipeline nodes labels Jul 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor
🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

@coderabbitai

coderabbitai Bot commented Jul 20, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Adds ArangoDB and HydraDB graph nodes, relocates vector/search nodes under store_* packages, implements provider clients and storage flows, and updates service registration, documentation, redirects, dependency loading, and test paths.

Changes

Graph database nodes

Layer / File(s) Summary
ArangoDB runtime, tools, and registration
nodes/src/nodes/graph_arango/*
Adds ArangoDB lifecycle management, schema reflection, AQL safety and EXPLAIN validation, LLM query generation, execution routing, dependency loading, and graph-node registration.
HydraDB client and tools
nodes/src/nodes/graph_hydradb/*
Adds a HydraDB REST client and tools for storing memories, recalling data, querying relations, and retrieving schema/status information.

Store package implementations

Layer / File(s) Summary
Provider store implementations
nodes/src/nodes/store_*/...
Adds or relocates Astra, Atlas, Chroma, Elasticsearch/OpenSearch, Milvus, Pinecone, Postgres, Qdrant, and Weaviate clients with search, ingestion, deletion, and rendering flows.
Lifecycle and package wiring
nodes/src/nodes/store_*/I*, nodes/src/nodes/store_*/__init__.py, nodes/src/nodes/store_*/services*.json
Adds global and instance lifecycle wiring, package exports, dependencies, and renamed service paths.
Documentation, redirects, and test paths
docs/*, packages/docs/redirects.ts, nodes/test/*, tools/dependabot-smoke/*
Updates node names, source links, legacy redirects, dynamic test loaders, skip configuration, and dependency smoke-test mappings.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

Suggested reviewers: jmaionchi, stepmikhaylov, rod-christensen, copilot

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 87.57% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main rename refactor for vector-store and graph-first database nodes.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/node-family-renames

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@stepmikhaylov

Copy link
Copy Markdown
Collaborator

How about using directories for organization instead of prefixes?

Instead of:

nodes/src/nodes/
     graph_arango/
     graph_hydradb/
     vectordb_astra/
     vectordb_atlas/
     ...

Just like tha:t

nodes/src/nodes/
    graph/
        arango/
        hydradb/
    vectordb/
        astra/
        atlas/
    ...

This would also free up the root nodes/src/nodes, which already contains over a hundred nodes/directories.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 32

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@nodes/src/nodes/graph_hydradb/requirements.txt`:
- Line 1: Update the requests dependency entry in requirements.txt to require
the safe minimum version requests>=2.34.2, preserving the existing dependency
declaration format.

In `@nodes/src/nodes/vectordb_astra/astra_db.py`:
- Around line 380-385: Update the object deletion logic in flush() to issue one
collection.delete_many call using an $in filter over list(objectIds.keys()),
replacing the per-object loop while preserving the existing no-objectIds guard.
- Around line 386-434: Update the local is_valid_vector filter in flush so the
schema control document is accepted regardless of its near-zero vector
magnitude, while retaining the existing magnitude threshold for all other
documents. Use the document’s metadata/objectId to identify the schema control
document and preserve its insertion through valid_docs.

In `@nodes/src/nodes/vectordb_astra/README.md`:
- Around line 81-99: Regenerate the params section in the renamed vectordb_astra
module using nodes:docs-generate instead of editing README.md manually. Ensure
the generated field keys and View source link reference vectordb_astra
consistently, while preserving the generated block markers and content outside
the block.
- Around line 1-3: Update the README heading and node title from astra_db to
vectordb_astra, keeping the existing description and documentation content
unchanged.

In `@nodes/src/nodes/vectordb_atlas/atlas.py`:
- Around line 392-441: Update render() to avoid preallocating text with
renderChunkSize; build the rendered chunk content dynamically from the documents
actually returned while preserving chunk ordering and pagination behavior. Keep
the existing metadata validation, offset bounds, callback invocation, and
termination conditions intact, and ensure sparse chunk IDs still produce the
correct concatenated output.
- Around line 270-276: Initialize or otherwise handle results in the
vector-search aggregation flow around self.collection.aggregate(pipeline) so an
OperationFailure cannot reach _convertToDocs with an unassigned value. Preserve
the original Atlas error by re-raising it after logging, or return an
appropriate empty result if that is the established method contract.

In `@nodes/src/nodes/vectordb_atlas/IGlobal.py`:
- Around line 56-62: Update the subKey construction in IGlobal using
self.store.database and self.store.collection so it extracts their name strings
rather than interpolating pymongo objects; prefer the store’s database_name and
collection_name fields when available, matching the name-based approach in
vectordb_astra/IGlobal.py, while preserving the host/database/collection format.
- Around line 83-86: Update validateConfig in IGlobal.py to remove the
api_key/apikey validation guard, since Store.__init__ and services.json do not
define or consume that field. Ensure validation continues to the existing host,
database, and collection checks when the field is absent.

In `@nodes/src/nodes/vectordb_atlas/README.md`:
- Line 1: Update the hand-authored Markdown heading in the README from “atlas”
to the renamed node name “vectordb_atlas.” Do not modify anything inside the
ROCKETRIDE:GENERATED:PARAMS block, including its generated links or content.

In `@nodes/src/nodes/vectordb_chroma/chroma.py`:
- Around line 438-462: Fix payload-size batching in the chunk loop by replacing
shallow sys.getsizeof(tmp_struct) accounting with an estimate based on the
actual content, embedding, and metadata payload sizes. Since sum_size already
includes the current item, change the payload threshold check to compare
sum_size directly against self.payload_limit while preserving the 500-item flush
condition.
- Around line 364-393: Update getPaths to filter chunk-zero records using the
stored metadata key chunkId instead of chunk, while preserving the existing
parent filter, query, and path-building behavior.
- Around line 227-262: Correct the score calculation in _convertToDocs so lower
Chroma distances produce higher similarity scores for cosine, l2, and ip
metrics. Update both similarity branches and ensure the score < 0.20 filtering
uses the corrected score ordering.
- Around line 483-529: Update markDeleted and markActive to handle
collectionObj.get’s parallel-array response rather than iterating the result
dictionary. Pair each returned ID with its metadata, set isDeleted to True or
False respectively, and call collectionObj.update using the supported ids= and
metadatas= arguments; preserve the existing collection-existence and empty-input
early returns.

In `@nodes/src/nodes/vectordb_chroma/README.md`:
- Around line 95-117: Regenerate the README’s generated documentation using the
nodes:docs-generate task so the Source link for the vectordb_chroma node targets
nodes/src/nodes/vectordb_chroma. Do not edit the generated block manually.

In `@nodes/src/nodes/vectordb_elasticsearch/elasticsearch_store.py`:
- Around line 382-383: In the semantic-search validation around
docFilter.offset, replace BaseException with Exception so the non-zero offset
error is handled by standard callers' exception paths. Preserve the existing
error message and validation behavior.
- Around line 583-591: Update the batching logic around the action accumulation
and payload-limit check to estimate each document’s serialized request size
rather than using sys.getsizeof(doc). Serialize the action using the same or
equivalent encoding used for the Elasticsearch bulk request, add its byte length
to sum_size, and retain the existing 500-action and payload-limit flush
behavior.

In `@nodes/src/nodes/vectordb_elasticsearch/IInstance.py`:
- Around line 314-320: Move the client.ensure_index_vector call out of the
per-document loop in the vector ingestion flow, placing it once before iteration
after validating that vector_dim is positive. Keep the existing embedding and
vector_dim skip behavior inside the loop, and use the constant vector_dim for
the single index initialization.

In `@nodes/src/nodes/vectordb_elasticsearch/requirements.txt`:
- Around line 1-3: Update the node dependency list in requirements.txt by adding
requests with a safe minimum constraint of requests>=2.34.2, while preserving
the existing elasticsearch, opensearch-py, and numpy dependencies.

In `@nodes/src/nodes/vectordb_milvus/IGlobal.py`:
- Around line 65-90: Update the token-based cloud branch in the validation
connection flow to construct the URI with the same https:// scheme and host/port
shape used by Store.__init__. Keep host_raw and port unchanged for local
connections, and continue passing the resulting URI to connections.connect.

In `@nodes/src/nodes/vectordb_milvus/milvus.py`:
- Around line 470-478: Update the delete error handling in addChunks so failures
from self.client.delete are logged and then re-raised, matching the behavior in
remove(). Do not continue inserting replacement chunks when deletion of existing
objectId chunks fails; preserve the current successful deletion and insertion
flow.
- Around line 663-674: Update the result-processing loop in render() to read
chunkId from each point’s meta JSON, matching getPaths(), instead of accessing a
non-existent top-level field. Add the upper-bound guard so chunks at or beyond
offset + self.renderChunkSize are skipped before indexing text, while preserving
the existing lower-bound check and rendering behavior for valid chunks.

In `@nodes/src/nodes/vectordb_pinecone/IGlobal.py`:
- Around line 177-182: Update the collection lookup and mode validation in the
Pinecone initialization flow to use IndexModel attributes: compare `index.name`
when finding `existing_collection`, then inspect `existing_collection.spec` for
serverless mode instead of calling `.get()` on the model. Preserve the existing
compatibility-check behavior.

In `@nodes/src/nodes/vectordb_pinecone/pinecone.py`:
- Around line 238-245: Update searchKeyword so it does not pass the already
converted filter dict to get(), since get() unconditionally calls _convertFilter
and expects a DocFilter. Preserve the keyword predicate by either passing the
original DocFilter and applying the content condition within the query path, or
adding an explicit raw-filter path to get() that supports the converted filter.
- Around line 288-290: Fix the threshold attribute naming in the Pinecone node:
rename the class attribute threshhold_search to threshold_search and update the
points_satisfy_threshold filter to read self.threshold_search, matching the
value initialized by __init__ so semantic search applies the configured score
threshold.

In `@nodes/src/nodes/vectordb_pinecone/README.md`:
- Around line 54-59: Reconcile the Pinecone profile default documentation with
the schema by updating the conflicting default references in the README,
including the profile table and the statement below it, so they consistently
identify pod-based as the default.

In `@nodes/src/nodes/vectordb_qdrant/qdrant.py`:
- Around line 140-149: Update Qdrant’s __del__ method to close the existing
self.client before clearing it, while safely handling the client being unset or
unavailable. Preserve the current attribute-reset behavior after cleanup.

In `@nodes/src/nodes/vectordb_weaviate/IGlobal.py`:
- Around line 71-154: Update validateConfig to determine cloud versus local
using the configured mode value, matching the mode selection used by
Store.__init__ in weaviate.py, instead of inferring it from host patterns.
Preserve the existing cloud HTTP probe and local SDK/gRPC readiness checks, but
route them according to the selected mode so save-time validation matches
runtime behavior.

In `@nodes/src/nodes/vectordb_weaviate/weaviate.py`:
- Around line 489-541: Update markDeleted and markActive to use Weaviate’s batch
API, consistent with addChunks, instead of calling collectionObj.data.update
once per point. Batch the matching point UUIDs with the appropriate isDeleted
value while preserving the existing collection checks, empty-input handling, and
filtering behavior.
- Around line 543-606: Update the fetch_objects call inside render to pass
limit=self.renderChunkSize, ensuring each chunk window requests the full
configured batch rather than Weaviate’s default page size. Preserve the existing
offset filtering, ordering, callback behavior, and termination logic.

In `@nodes/test/vectordb_astra/test_convert_filter.py`:
- Line 15: Update the Store import in test_convert_filter.py to use the
relocated nodes.vectordb_astra module path, matching the package structure used
by the moved implementation. Remove the stale nodes.astra_db.astra_db reference
so the test can import Store and exercise _convertFilter.

In `@nodes/test/vectordb_milvus/test_addchunks_return.py`:
- Around line 19-23: Update the import inside
test_milvus_addchunks_returns_none_when_create_collection_fails to reference the
current Milvus module path after the package move, removing the stale
nodes.milvus.milvus target so the test can run when its skip is removed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 48b786df-7ae7-47b9-a373-642deb5e5c99

📥 Commits

Reviewing files that changed from the base of the PR and between 4a09a43 and 75d3f81.

⛔ Files ignored due to path filters (11)
  • nodes/src/nodes/graph_arango/arango.svg is excluded by !**/*.svg
  • nodes/src/nodes/graph_hydradb/hydradb.svg is excluded by !**/*.svg
  • nodes/src/nodes/vectordb_astra/astra_db.svg is excluded by !**/*.svg
  • nodes/src/nodes/vectordb_atlas/mongodb.svg is excluded by !**/*.svg
  • nodes/src/nodes/vectordb_chroma/chroma.svg is excluded by !**/*.svg
  • nodes/src/nodes/vectordb_elasticsearch/elasticsearch.svg is excluded by !**/*.svg
  • nodes/src/nodes/vectordb_elasticsearch/opensearch.svg is excluded by !**/*.svg
  • nodes/src/nodes/vectordb_milvus/milvus.svg is excluded by !**/*.svg
  • nodes/src/nodes/vectordb_pinecone/pinecone.svg is excluded by !**/*.svg
  • nodes/src/nodes/vectordb_qdrant/qdrant.svg is excluded by !**/*.svg
  • nodes/src/nodes/vectordb_weaviate/weaviate.svg is excluded by !**/*.svg
📒 Files selected for processing (106)
  • nodes/src/nodes/graph_arango/IGlobal.py
  • nodes/src/nodes/graph_arango/IInstance.py
  • nodes/src/nodes/graph_arango/README.md
  • nodes/src/nodes/graph_arango/__init__.py
  • nodes/src/nodes/graph_arango/requirements.txt
  • nodes/src/nodes/graph_arango/services.json
  • nodes/src/nodes/graph_arango/utils.py
  • nodes/src/nodes/graph_hydradb/IGlobal.py
  • nodes/src/nodes/graph_hydradb/IInstance.py
  • nodes/src/nodes/graph_hydradb/README.md
  • nodes/src/nodes/graph_hydradb/__init__.py
  • nodes/src/nodes/graph_hydradb/hydradb_client.py
  • nodes/src/nodes/graph_hydradb/requirements.txt
  • nodes/src/nodes/graph_hydradb/services.json
  • nodes/src/nodes/vectordb_astra/IEndpoint.py
  • nodes/src/nodes/vectordb_astra/IGlobal.py
  • nodes/src/nodes/vectordb_astra/IInstance.py
  • nodes/src/nodes/vectordb_astra/README.md
  • nodes/src/nodes/vectordb_astra/__init__.py
  • nodes/src/nodes/vectordb_astra/astra_db.py
  • nodes/src/nodes/vectordb_astra/requirements.txt
  • nodes/src/nodes/vectordb_astra/services.json
  • nodes/src/nodes/vectordb_atlas/IEndpoint.py
  • nodes/src/nodes/vectordb_atlas/IGlobal.py
  • nodes/src/nodes/vectordb_atlas/IInstance.py
  • nodes/src/nodes/vectordb_atlas/README.md
  • nodes/src/nodes/vectordb_atlas/__init__.py
  • nodes/src/nodes/vectordb_atlas/atlas.py
  • nodes/src/nodes/vectordb_atlas/requirements.txt
  • nodes/src/nodes/vectordb_atlas/services.json
  • nodes/src/nodes/vectordb_chroma/IEndpoint.py
  • nodes/src/nodes/vectordb_chroma/IGlobal.py
  • nodes/src/nodes/vectordb_chroma/IInstance.py
  • nodes/src/nodes/vectordb_chroma/README.md
  • nodes/src/nodes/vectordb_chroma/__init__.py
  • nodes/src/nodes/vectordb_chroma/chroma.py
  • nodes/src/nodes/vectordb_chroma/requirements.txt
  • nodes/src/nodes/vectordb_chroma/services.json
  • nodes/src/nodes/vectordb_elasticsearch/IEndpoint.py
  • nodes/src/nodes/vectordb_elasticsearch/IGlobal.py
  • nodes/src/nodes/vectordb_elasticsearch/IInstance.py
  • nodes/src/nodes/vectordb_elasticsearch/README.md
  • nodes/src/nodes/vectordb_elasticsearch/__init__.py
  • nodes/src/nodes/vectordb_elasticsearch/constants.py
  • nodes/src/nodes/vectordb_elasticsearch/elasticsearch/README.md
  • nodes/src/nodes/vectordb_elasticsearch/elasticsearch_store.py
  • nodes/src/nodes/vectordb_elasticsearch/opensearch/README.md
  • nodes/src/nodes/vectordb_elasticsearch/opensearch_client.py
  • nodes/src/nodes/vectordb_elasticsearch/requirements.txt
  • nodes/src/nodes/vectordb_elasticsearch/services.elasticsearch.json
  • nodes/src/nodes/vectordb_elasticsearch/services.opensearch.json
  • nodes/src/nodes/vectordb_milvus/IEndpoint.py
  • nodes/src/nodes/vectordb_milvus/IGlobal.py
  • nodes/src/nodes/vectordb_milvus/IInstance.py
  • nodes/src/nodes/vectordb_milvus/README.md
  • nodes/src/nodes/vectordb_milvus/__init__.py
  • nodes/src/nodes/vectordb_milvus/milvus.py
  • nodes/src/nodes/vectordb_milvus/requirements.txt
  • nodes/src/nodes/vectordb_milvus/services.json
  • nodes/src/nodes/vectordb_pinecone/IEndpoint.py
  • nodes/src/nodes/vectordb_pinecone/IGlobal.py
  • nodes/src/nodes/vectordb_pinecone/IInstance.py
  • nodes/src/nodes/vectordb_pinecone/README.md
  • nodes/src/nodes/vectordb_pinecone/__init__.py
  • nodes/src/nodes/vectordb_pinecone/pinecone.py
  • nodes/src/nodes/vectordb_pinecone/requirements.txt
  • nodes/src/nodes/vectordb_pinecone/services.json
  • nodes/src/nodes/vectordb_qdrant/IEndpoint.py
  • nodes/src/nodes/vectordb_qdrant/IGlobal.py
  • nodes/src/nodes/vectordb_qdrant/IInstance.py
  • nodes/src/nodes/vectordb_qdrant/README.md
  • nodes/src/nodes/vectordb_qdrant/__init__.py
  • nodes/src/nodes/vectordb_qdrant/qdrant.py
  • nodes/src/nodes/vectordb_qdrant/requirements.txt
  • nodes/src/nodes/vectordb_qdrant/services.json
  • nodes/src/nodes/vectordb_weaviate/IEndpoint.py
  • nodes/src/nodes/vectordb_weaviate/IGlobal.py
  • nodes/src/nodes/vectordb_weaviate/IInstance.py
  • nodes/src/nodes/vectordb_weaviate/README.md
  • nodes/src/nodes/vectordb_weaviate/__init__.py
  • nodes/src/nodes/vectordb_weaviate/requirements.txt
  • nodes/src/nodes/vectordb_weaviate/services.json
  • nodes/src/nodes/vectordb_weaviate/weaviate.py
  • nodes/test/conftest.py
  • nodes/test/ocr/test_model_server_ocr.py
  • nodes/test/test_graph_arango.py
  • nodes/test/test_graph_hydradb.py
  • nodes/test/vectordb_astra/__init__.py
  • nodes/test/vectordb_astra/test_convert_filter.py
  • nodes/test/vectordb_atlas/__init__.py
  • nodes/test/vectordb_atlas/test_isdeleted_filter.py
  • nodes/test/vectordb_chroma/__init__.py
  • nodes/test/vectordb_chroma/test_coerce_port.py
  • nodes/test/vectordb_chroma/test_convert_filter_explicit_none.py
  • nodes/test/vectordb_milvus/__init__.py
  • nodes/test/vectordb_milvus/test_addchunks_return.py
  • nodes/test/vectordb_milvus/test_filter_injection.py
  • nodes/test/vectordb_milvus/test_search_semantic.py
  • nodes/test/vectordb_pinecone/__init__.py
  • nodes/test/vectordb_pinecone/test_record_updates.py
  • nodes/test/vectordb_qdrant/__init__.py
  • nodes/test/vectordb_qdrant/test_mark_deleted.py
  • nodes/test/vectordb_qdrant/test_search_threshold.py
  • nodes/test/vectordb_weaviate/__init__.py
  • nodes/test/vectordb_weaviate/test_record_updates.py
  • packages/docs/redirects.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

🛑 Comments failed to post (32)
nodes/src/nodes/graph_hydradb/requirements.txt (1)

1-1: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

Pin requests to a safe minimum version.

To avoid resolving vulnerable older versions of requests and to maintain accurate dependency/vulnerability scans, please pin requests to a safe minimum version. Based on learnings, it is recommended to use requests>=2.34.2 in the node requirements.txt files.

🔒️ Proposed fix to pin requests
-requests
+requests>=2.34.2
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

requests>=2.34.2
🧰 Tools
🪛 OSV Scanner (2.4.0)

[HIGH] 1-1: requests 2.9.2: undefined

(PYSEC-2018-28)


[HIGH] 1-1: requests 2.9.2: undefined

(PYSEC-2023-74)


[HIGH] 1-1: requests 2.9.2: Requests vulnerable to .netrc credentials leak via malicious URLs

(PYSEC-2026-1872)


[HIGH] 1-1: requests 2.9.2: Requests Session object does not verify requests after making first request with verify=False

(PYSEC-2026-1873)


[HIGH] 1-1: requests 2.9.2: undefined

(PYSEC-2026-2275)


[HIGH] 1-1: requests 2.9.2: Requests vulnerable to .netrc credentials leak via malicious URLs

(GHSA-9hjg-9r4m-mvj7)


[HIGH] 1-1: requests 2.9.2: Requests Session object does not verify requests after making first request with verify=False

(GHSA-9wx4-h78v-vm56)


[HIGH] 1-1: requests 2.9.2: Requests has Insecure Temp File Reuse in its extract_zipped_paths() utility function

(GHSA-gc5v-m9x4-r6x2)


[HIGH] 1-1: requests 2.9.2: Unintended leak of Proxy-Authorization header in requests

(GHSA-j8r2-6x86-q33q)


[HIGH] 1-1: requests 2.9.2: Insufficiently Protected Credentials in Requests

(GHSA-x84v-xcm2-53pg)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/graph_hydradb/requirements.txt` at line 1, Update the
requests dependency entry in requirements.txt to require the safe minimum
version requests>=2.34.2, preserving the existing dependency declaration format.

Source: Learnings

nodes/src/nodes/vectordb_astra/astra_db.py (2)

380-385: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Consider batching the per-object deletes into a single call.

flush() issues one delete_many({'meta.objectId': objectId}) call per objectId in the batch instead of a single delete_many({'meta.objectId': {'$in': list(objectIds.keys())}}). For batches touching many distinct objects this is N round-trips instead of 1.

♻️ Proposed fix
-            if len(objectIds):
-                # Delete all documents with these object IDs
-                for objectId in objectIds.keys():
-                    collection.delete_many({'meta.objectId': objectId})
+            if len(objectIds):
+                # Delete all documents with these object IDs in one round-trip
+                collection.delete_many({'meta.objectId': {'$in': list(objectIds.keys())}})
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

            # First, delete existing documents if we have objectIds to delete
            if len(objectIds):
                # Delete all documents with these object IDs in one round-trip
                collection.delete_many({'meta.objectId': {'$in': list(objectIds.keys())}})
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_astra/astra_db.py` around lines 380 - 385, Update
the object deletion logic in flush() to issue one collection.delete_many call
using an $in filter over list(objectIds.keys()), replacing the per-object loop
while preserving the existing no-objectIds guard.

386-434: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Schema control-document vector is likely always filtered out by is_valid_vector, breaking control-doc persistence.

The zero-vector workaround (line 417-418) sets the schema control doc's embedding to [1e-8] * len(embedding), but flush()'s is_valid_vector (line 386-388) requires magnitude > 1e-6. For any realistic embedding dimension (e.g. 384–3072), magnitude = 1e-8 * sqrt(dim) stays well below 1e-6 (e.g. ≈3.9e-7 for dim=1536), so the control document is silently excluded from valid_docs and never inserted. Since doesCollectionExist()'s docstring explicitly relies on a "control document" for validation, this defeats the very workaround meant to let the schema doc pass the ingest path, causing the base class's collection validation to always treat a freshly created collection as missing its control document.

🐛 Proposed fix: exempt the schema control document from the magnitude filter
-            def is_valid_vector(vector):
-                magnitude = sum(x * x for x in vector) ** 0.5
-                return magnitude > 1e-6
-
-            valid_docs = [doc for doc in documents if is_valid_vector(doc.get('$vector', [1]))]
+            def is_valid_vector(doc):
+                # The schema control document intentionally uses a near-zero
+                # vector; never drop it based on magnitude.
+                if doc.get('meta', {}).get('objectId') == 'schema':
+                    return True
+                vector = doc.get('$vector', [1])
+                magnitude = sum(x * x for x in vector) ** 0.5
+                return magnitude > 1e-6
+
+            valid_docs = [doc for doc in documents if is_valid_vector(doc)]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

            def is_valid_vector(doc):
                # The schema control document intentionally uses a near-zero
                # vector; never drop it based on magnitude.
                if doc.get('meta', {}).get('objectId') == 'schema':
                    return True
                vector = doc.get('$vector', [1])
                magnitude = sum(x * x for x in vector) ** 0.5
                return magnitude > 1e-6

            valid_docs = [doc for doc in documents if is_valid_vector(doc)]

            # Insert the new documents
            if len(valid_docs):
                collection.insert_many(valid_docs)

            # Clear them
            objectIds = {}
            documents = []

        # For each document, check if we need to delete existing chunks
        for chunk in chunks:
            # If we are writing chunk 0, then delete the object
            if not chunk.metadata.chunkId:
                # Save this object id to be deleted
                objectIds[chunk.metadata.objectId] = True

        # For each document, prepare the data for insertion
        for chunk in chunks:
            # Get the embedding
            embedding = chunk.embedding

            # If we do not have an embedding
            if embedding is None:
                raise Exception('No embedding in document')

            # Handle control document with zero vectors
            if chunk.metadata.objectId == 'schema' and all(x == 0 for x in embedding):
                embedding = [1e-8] * len(embedding)

            # Create the document structure for AstraDB
            document = {
                '_id': str(uuid4()),
                '$vector': DataAPIVector(embedding),
                'content': chunk.page_content,
                'meta': chunk.metadata.__dict__,  # Convert metadata to dict
            }

            documents.append(document)

            if len(documents) >= 500:
                flush()

        # Flush any remaining documents
        flush()
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_astra/astra_db.py` around lines 386 - 434, Update
the local is_valid_vector filter in flush so the schema control document is
accepted regardless of its near-zero vector magnitude, while retaining the
existing magnitude threshold for all other documents. Use the document’s
metadata/objectId to identify the schema control document and preserve its
insertion through valid_docs.
nodes/src/nodes/vectordb_astra/README.md (2)

1-3: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Title still references the old node name.

The heading (# astra_db) and node title weren't updated to vectordb_astra as part of this rename, unlike the folder/module path itself.

✏️ Proposed fix
-# astra_db
+# vectordb_astra
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

# vectordb_astra

A RocketRide store node that persists embedded documents in DataStax Astra DB and retrieves them by semantic or keyword search.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_astra/README.md` around lines 1 - 3, Update the
README heading and node title from astra_db to vectordb_astra, keeping the
existing description and documentation content unchanged.

81-99: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Generated params section still references the pre-rename node name/path.

Field keys (astra_db.api_endpoint, astra_db.application_token, astra_db.provider) and the "View source" link (.../nodes/src/nodes/astra_db) are stale post-rename. Per path instructions, this block must not be hand-edited (it's regenerated by nodes:docs-generate), so the fix is to re-run the doc generator against the renamed vectordb_astra module rather than editing these lines directly.

As per path instructions, nodes/src/nodes/*/README.md: "do not hand-edit content between <!-- ROCKETRIDE:GENERATED:PARAMS START --> and <!-- ROCKETRIDE:GENERATED:PARAMS END --> as it is regenerated by nodes:docs-generate."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_astra/README.md` around lines 81 - 99, Regenerate
the params section in the renamed vectordb_astra module using
nodes:docs-generate instead of editing README.md manually. Ensure the generated
field keys and View source link reference vectordb_astra consistently, while
preserving the generated block markers and content outside the block.

Source: Path instructions

nodes/src/nodes/vectordb_atlas/atlas.py (2)

270-276: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

results may be unbound if the vector-search aggregation fails.

If self.collection.aggregate(pipeline) raises OperationFailure, the except block only logs the error and falls through — results is never assigned, so self._convertToDocs(results, include_score=True) on line 276 raises UnboundLocalError instead of surfacing the original Atlas error.

🐛 Proposed fix
         try:
             results = list(self.collection.aggregate(pipeline))
             logger.info(f'Results: {results}')
         except OperationFailure as e:
             logger.error(f'OperationFailure: {e}')
+            return []

         return self._convertToDocs(results, include_score=True)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        try:
            results = list(self.collection.aggregate(pipeline))
            logger.info(f'Results: {results}')
        except OperationFailure as e:
            logger.error(f'OperationFailure: {e}')
            return []

        return self._convertToDocs(results, include_score=True)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_atlas/atlas.py` around lines 270 - 276, Initialize
or otherwise handle results in the vector-search aggregation flow around
self.collection.aggregate(pipeline) so an OperationFailure cannot reach
_convertToDocs with an unassigned value. Preserve the original Atlas error by
re-raising it after logging, or return an appropriate empty result if that is
the established method contract.

392-441: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

render() preallocates a list sized to renderChunkSize (default ~33.5M) on every call/iteration.

text = [''] * self.renderChunkSize with the default renderChunkSize = 32 * 1024 * 1024 allocates a ~33.5-million-element list even for objects with a single chunk, since this happens unconditionally before any documents are inspected. This is a significant memory/latency cost on what looks like a routine rendering path. The same pattern appears in sibling providers (astra_db.py doesn't have it, but chroma.py/milvus.py/pinecone.py/weaviate.py do), suggesting renderChunkSize's intended semantics (payload byte-size limit vs. chunk-count page size) may be conflated across the codebase.

🧰 Tools
🪛 ast-grep (0.44.1)

[warning] 403-403: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: self.collection.find(filter_conditions)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').

(xpath-injection-python)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_atlas/atlas.py` around lines 392 - 441, Update
render() to avoid preallocating text with renderChunkSize; build the rendered
chunk content dynamically from the documents actually returned while preserving
chunk ordering and pagination behavior. Keep the existing metadata validation,
offset bounds, callback invocation, and termination conditions intact, and
ensure sparse chunk IDs still produce the correct concatenated output.
nodes/src/nodes/vectordb_atlas/IGlobal.py (2)

56-62: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win

subKey is built from pymongo Database/Collection objects, not their names.

self.store.database and self.store.collection are pymongo Database/Collection instances (per atlas.py's self.database = self.client[self.database_name] / self.database[self.collection_name]), not name strings. Interpolating them into subKey yields their verbose repr() (e.g. Database(MongoClient(host=[...], ...), 'testdb')) instead of the intended host/database/collection identifier. Contrast with vectordb_astra/IGlobal.py, which explicitly guards against this by preferring collection_name over collection.

🐛 Proposed fix
             # Get the info about our store
-            database = self.store.database
-            collection = self.store.collection
+            database = self.store.database_name
+            collection = self.store.collection_name
             host = self.store.host
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

            # Get the info about our store
            database = self.store.database_name
            collection = self.store.collection_name
            host = self.store.host

            # Format it into a subKey
            subKey = f'{host}/{database}/{collection}'
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_atlas/IGlobal.py` around lines 56 - 62, Update the
subKey construction in IGlobal using self.store.database and
self.store.collection so it extracts their name strings rather than
interpolating pymongo objects; prefer the store’s database_name and
collection_name fields when available, matching the name-based approach in
vectordb_astra/IGlobal.py, while preserving the host/database/collection format.

83-86: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== file list ==\n'
git ls-files | rg '(^|/)(IGlobal\.py|atlas\.py|services\.json|README\.md|.*schema.*|.*services.*)$' || true

printf '\n== locate relevant symbols ==\n'
rg -n --hidden --glob '!**/.git/**' 'validateConfig|apikey|atlas.provider|vector.cloud.host|vector.database|Store\.__init__|class Store' nodes src . || true

printf '\n== inspect IGlobal.py ==\n'
sed -n '1,180p' nodes/src/nodes/vectordb_atlas/IGlobal.py

printf '\n== inspect atlas.py ==\n'
sed -n '1,220p' nodes/src/nodes/vectordb_atlas/atlas.py

printf '\n== inspect services.json candidates ==\n'
fd -a 'services\.json$|schema' . || true

Repository: rocketride-org/rocketride-server

Length of output: 50389


🏁 Script executed:

#!/bin/bash
set -euo pipefail
sed -n '1,180p' nodes/src/nodes/vectordb_atlas/IGlobal.py
printf '\n----\n'
sed -n '1,220p' nodes/src/nodes/vectordb_atlas/atlas.py
printf '\n----\n'
rg -n 'apikey|atlas.provider|vector.cloud.host|vector.database|validateConfig|services\.json' nodes/src/nodes/vectordb_atlas . --glob '!**/.git/**'

Repository: rocketride-org/rocketride-server

Length of output: 50389


Remove the apikey guard or add the field to the schema — nodes/src/nodes/vectordb_atlas/services.json exposes only atlas.provider, vector.cloud.host, and vector.database, and Store.__init__ never reads apikey. As written, validateConfig exits before host/database/collection checks for any config without that field.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_atlas/IGlobal.py` around lines 83 - 86, Update
validateConfig in IGlobal.py to remove the api_key/apikey validation guard,
since Store.__init__ and services.json do not define or consume that field.
Ensure validation continues to the existing host, database, and collection
checks when the field is absent.
nodes/src/nodes/vectordb_atlas/README.md (1)

1-1: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Title still says # atlas; should reflect the renamed node.

This heading is outside the <!-- ROCKETRIDE:GENERATED:PARAMS --> block, so it's hand-authored and wasn't updated along with the folder/module rename to vectordb_atlas.

📝 Proposed fix
-# atlas
+# vectordb_atlas

As per path instructions, the generated params block (including the stale "View source" link at line 76) should not be hand-edited, since it is regenerated by nodes:docs-generate. Based on path instructions: "Documentation must be co-located with the code it documents ... do not hand-edit content between <!-- ROCKETRIDE:GENERATED:PARAMS START --> and <!-- ROCKETRIDE:GENERATED:PARAMS END --> as it is regenerated by nodes:docs-generate."

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

# vectordb_atlas
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_atlas/README.md` at line 1, Update the hand-authored
Markdown heading in the README from “atlas” to the renamed node name
“vectordb_atlas.” Do not modify anything inside the ROCKETRIDE:GENERATED:PARAMS
block, including its generated links or content.

Source: Path instructions

nodes/src/nodes/vectordb_chroma/chroma.py (4)

227-262: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

chromadb query distances cosine l2 ip meaning range

💡 Result:

In ChromaDB, the distance metric defines how the similarity between vectors is calculated, with lower values indicating higher similarity [1][2]. You can configure this at the collection creation level using the hnsw.space parameter [1][3]. The available options are l2, cosine, and ip [1][4][3]. Squared L2 (l2) - Definition: Squared Euclidean distance [1][4]. - Equation: d = Σ(Ai - Bi)² [4]. - Range: Starts at 0 (identical) and increases toward positive infinity [2]. - Usage: Measures absolute geometric distance, making it suitable when true spatial proximity is required [4]. It is the default metric in ChromaDB [1][4]. Cosine Similarity (cosine) - Definition: Measures the cosine distance between two vectors (often used for text embeddings) [4]. - Equation: d = 1.0 - (Σ(Ai * Bi) / (√Σ(Ai²) * √Σ(Bi²))) [4]. - Range: Ranges from 0 to 2. A distance of 0 means the vectors are identical (cosine similarity = 1), and a distance of 2 means the vectors are opposite (cosine similarity = -1). - Usage: Focuses on the angle/direction between vectors while ignoring their magnitude [4]. Inner Product (ip) - Definition: Measures the inner product distance [4]. - Equation: d = 1.0 - Σ(Ai * Bi) [4]. - Range: Does not have a fixed upper bound; it depends on the magnitude of the vectors. - Usage: Often used in recommendation systems where both vector alignment and magnitude are important [4]. Important Note: L2 and IP are sensitive to the magnitude of the vectors [2]. It is recommended to normalize your embeddings before adding them to ChromaDB if you are using these metrics [2]. Note that the distance metric cannot be changed after a collection has been created [1][4][3].

Citations:


Score mapping is reversed for Chroma distances.
Chroma reports lower distances for better matches across cosine, l2, and ip, but both branches here turn larger distances into larger scores. That ranks worse hits above better ones and makes the score < 0.20 cutoff ineffective.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_chroma/chroma.py` around lines 227 - 262, Correct
the score calculation in _convertToDocs so lower Chroma distances produce higher
similarity scores for cosine, l2, and ip metrics. Update both similarity
branches and ensure the score < 0.20 filtering uses the corrected score
ordering.

364-393: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

getPaths filters on 'chunk' instead of 'chunkId'.

Every other filter in this file (_convertFilter, render) consistently keys chunk-index metadata as 'chunkId'. getPaths's base filter uses filter_dict = {'chunk': 0} — a field that doesn't exist in stored metadata, so the ChromaDB where filter will not match chunk-0 records, and getPaths will effectively always return {}.

🐛 Proposed fix
-        filter_dict = {'chunk': 0}
+        filter_dict = {'chunkId': {'$eq': 0}}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def getPaths(self, parent: str | None = None, offset: int = 0, limit: int = 1000) -> Dict[str, str]:
        """
        Retrieve unique parent paths.
        """
        # If the collection does not exists, by definition there are
        # no paths to return
        if not self.doesCollectionExist():
            return {}

        # Base filter: Only chunk 0
        filter_dict = {'chunkId': {'$eq': 0}}

        # If parent specified, add condition
        if parent is not None:
            filter_dict['parent'] = {'$eq': json.dumps(parent, ensure_ascii=False)}

        # Perform the query
        results = self.collectionObj.get(where=filter_dict, limit=limit, offset=offset)

        # Build paths dictionary
        paths = {}
        for i in range(len(results['ids'])):
            metadata = results['metadatas'][i]
            parent_id = metadata.get('parent', None)
            object_id = metadata.get('objectId', None)

            if parent_id and object_id:
                paths[parent_id] = object_id

        return paths
🧰 Tools
🪛 ast-grep (0.44.1)

[info] 377-377: use jsonify instead of json.dumps for JSON output
Context: json.dumps(parent, ensure_ascii=False)
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_chroma/chroma.py` around lines 364 - 393, Update
getPaths to filter chunk-zero records using the stored metadata key chunkId
instead of chunk, while preserving the existing parent filter, query, and
path-building behavior.

438-462: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win

Payload-size batching is effectively broken.

sys.getsizeof(tmp_struct) (line 456) only measures the shallow size of the dict container itself, not the nested content string, meta dict, or vector list it references — so cur_size stays roughly constant (a few hundred bytes) regardless of actual embedding/document size, making the sum_size + cur_size > self.payload_limit check almost never trigger. Large batches of sizeable embeddings/content can therefore accumulate well past payload_limit before a flush happens, relying solely on the 500-item count cap. Separately, sum_size is already incremented by cur_size on line 457 before the check on line 459 adds cur_size again, double-counting the current item in the threshold comparison.

Consider computing size from the actual serialized payload (e.g., len(content.encode()), len(embedding) * 4 for float32, plus a JSON-encoded metadata estimate) rather than sys.getsizeof on the wrapper dict, and drop the redundant + cur_size in the comparison since sum_size already includes it.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_chroma/chroma.py` around lines 438 - 462, Fix
payload-size batching in the chunk loop by replacing shallow
sys.getsizeof(tmp_struct) accounting with an estimate based on the actual
content, embedding, and metadata payload sizes. Since sum_size already includes
the current item, change the payload threshold check to compare sum_size
directly against self.payload_limit while preserving the 500-item flush
condition.

483-529: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

chromadb python client Collection.update method signature ids metadatas where

💡 Result:

The ChromaDB Python client Collection.update method does not accept a where clause parameter [1]. The where parameter is used for filtering during queries or deletions (e.g., in .get(), .query(), or .delete()), but not for updating specific records [2][3]. The Collection.update method is designed to update existing records by their specific IDs [2][1]. The method signature is as follows [1]: collection.update( ids: OneOrMany[ID], embeddings: Optional[OneOrMany[Embedding]] = None, metadatas: Optional[OneOrMany[Metadata]] = None, documents: Optional[OneOrMany[Document]] = None, images: Optional[OneOrMany[Image]] = None, uris: Optional[OneOrMany[URI]] = None) Key points: - ids: Required. These are the unique identifiers of the records you wish to update [2][1]. If an ID provided does not exist in the collection, an error will be logged and the update for that specific record will be ignored [4]. - metadatas: Optional. You can provide new metadata dictionaries to replace or update the existing metadata for the specified IDs [2][1]. - No where clause: Because this method targets records by their explicit ID, it does not support filtering records via a where clause [2][1]. To update records based on a filter, you must first retrieve the IDs using a method like .get(where=...) and then pass those IDs to .update() [2][1]. When providing multiple values (like a list of IDs and a corresponding list of metadata), they must be of the same length, as the data is processed in a columnar format where the index of the element corresponds to the record [2][1].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the target file around the reported lines.
file='nodes/src/nodes/vectordb_chroma/chroma.py'
wc -l "$file"
sed -n '430,560p' "$file"

# Find any other Chroma collection update/get usages for context.
rg -n "collectionObj\.(get|update)\(" nodes/src/nodes/vectordb_chroma -n

# Inspect package/dependency pins that may indicate chromadb version.
rg -n "chromadb|chroma" -g 'pyproject.toml' -g 'requirements*.txt' -g 'poetry.lock' -g 'uv.lock' -g 'Pipfile.lock' -g 'setup.cfg' -g 'setup.py' -g 'package.json' -g 'environment.yml' .

Repository: rocketride-org/rocketride-server

Length of output: 5932


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file='nodes/src/nodes/vectordb_chroma/chroma.py'

# Show the class definition and how collectionObj is initialized.
rg -n "collectionObj|chromadb|Collection" "$file"
sed -n '1,220p' "$file"

# Look for any local helper that might wrap Chroma's Collection methods.
rg -n "def .*update|def .*get|class .*Collection|collectionObj =" nodes/src/nodes/vectordb_chroma

Repository: rocketride-org/rocketride-server

Length of output: 12597


Fix markDeleted/markActive to update records by ID.
self.collectionObj.get(...) returns a dict of parallel arrays, so iterating results walks keys, not records; update() also expects ids= and metadatas=, not where=/new_metadata=. Both methods fail on any non-empty objectIds.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_chroma/chroma.py` around lines 483 - 529, Update
markDeleted and markActive to handle collectionObj.get’s parallel-array response
rather than iterating the result dictionary. Pair each returned ID with its
metadata, set isDeleted to True or False respectively, and call
collectionObj.update using the supported ids= and metadatas= arguments; preserve
the existing collection-existence and empty-input early returns.
nodes/src/nodes/vectordb_chroma/README.md (1)

95-117: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- README excerpt ---'
sed -n '90,125p' nodes/src/nodes/vectordb_chroma/README.md

echo
echo '--- node directories ---'
find nodes/src/nodes -maxdepth 2 -type d | sort

echo
echo '--- possible chroma/vectordb_chroma files ---'
find nodes/src/nodes -maxdepth 2 \( -path '*/chroma*' -o -path '*/vectordb_chroma*' \) | sort

Repository: rocketride-org/rocketride-server

Length of output: 7907


🏁 Script executed:

#!/bin/bash
set -euo pipefail

sed -n '1,220p' nodes/src/nodes/vectordb_chroma/README.md | nl -ba | sed -n '90,125p'

echo
echo '--- directories/files ---'
find nodes/src/nodes -maxdepth 2 \( -path '*/chroma*' -o -path '*/vectordb_chroma*' \) -print | sort

echo
echo '--- docs-generation hints ---'
rg -n "ROCKETRIDE:GENERATED:PARAMS|View source|vectordb_chroma|nodes/chroma" nodes/src/nodes -S

Repository: rocketride-org/rocketride-server

Length of output: 210


Regenerate the generated Source link. It still points to nodes/src/nodes/chroma; rerun nodes:docs-generate so the README links to nodes/src/nodes/vectordb_chroma instead of editing the generated block by hand.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_chroma/README.md` around lines 95 - 117, Regenerate
the README’s generated documentation using the nodes:docs-generate task so the
Source link for the vectordb_chroma node targets
nodes/src/nodes/vectordb_chroma. Do not edit the generated block manually.

Source: Path instructions

nodes/src/nodes/vectordb_elasticsearch/elasticsearch_store.py (2)

382-383: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Raise Exception, not BaseException.

BaseException bypasses callers' except Exception handlers and can crash the pipeline instead of being handled/formatted like other store errors.

🐛 Proposed fix
-        if docFilter.offset:
-            raise BaseException('Non-zero offset is not supported in semantic searching')
+        if docFilter.offset:
+            raise Exception('Non-zero offset is not supported in semantic searching')
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        if docFilter.offset:
            raise Exception('Non-zero offset is not supported in semantic searching')
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_elasticsearch/elasticsearch_store.py` around lines
382 - 383, In the semantic-search validation around docFilter.offset, replace
BaseException with Exception so the non-zero offset error is handled by standard
callers' exception paths. Preserve the existing error message and validation
behavior.

583-591: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win

sys.getsizeof(doc) does not measure serialized payload size, so the payload_limit guard is effectively dead.

sys.getsizeof returns only the shallow size of the outer dict (a few hundred bytes); it does not account for the nested embedding list, page_content, or meta. As a result sum_size stays tiny and the sum_size > self.payload_limit branch never triggers — only the 500-action count flush applies. For large chunks this can let a single bulk request exceed Elasticsearch's http.max_content_length.

Approximate the real payload size instead (e.g. via serialization length):

♻️ Suggested size estimate
+import json
...
-            cur_size = sys.getsizeof(doc)
+            cur_size = len(json.dumps(doc['_source'], default=str).encode('utf-8'))
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

            cur_size = len(json.dumps(doc['_source'], default=str).encode('utf-8'))
            sum_size += cur_size
            actions.append(doc)

            # Flush if we hit batch size or payload limit
            if (len(actions) >= 500) or (sum_size > self.payload_limit):
                _run_bulk(actions)
                actions = []
                sum_size = 0
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_elasticsearch/elasticsearch_store.py` around lines
583 - 591, Update the batching logic around the action accumulation and
payload-limit check to estimate each document’s serialized request size rather
than using sys.getsizeof(doc). Serialize the action using the same or equivalent
encoding used for the Elasticsearch bulk request, add its byte length to
sum_size, and retain the existing 500-action and payload-limit flush behavior.
nodes/src/nodes/vectordb_elasticsearch/IInstance.py (1)

314-320: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Hoist ensure_index_vector out of the per-document loop.

ensure_index_vector runs indices.exists + get_mapping on every call, so invoking it once per document adds a network round-trip per chunk during ingestion. vector_dim is constant across the loop, so ensure the index once before iterating.

♻️ Suggested restructure
         vector_dim = self._os_get_vector_dim()
         debug(f'writeDocuments ingest count={len(documents)} index={index} mode=vstore')
+        if vector_dim <= 0:
+            debug('Vector mode missing vector_dim; cannot index')
+            return
+        client.ensure_index_vector(index=index, dimension=vector_dim)
 
         for doc in documents:
             ...
-            if vector_dim <= 0:
-                debug('Vector mode missing vector_dim; cannot index')
-                continue
-            client.ensure_index_vector(index=index, dimension=vector_dim)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        vector_dim = self._os_get_vector_dim()
        debug(f'writeDocuments ingest count={len(documents)} index={index} mode=vstore')
        if vector_dim <= 0:
            debug('Vector mode missing vector_dim; cannot index')
            return
        client.ensure_index_vector(index=index, dimension=vector_dim)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_elasticsearch/IInstance.py` around lines 314 - 320,
Move the client.ensure_index_vector call out of the per-document loop in the
vector ingestion flow, placing it once before iteration after validating that
vector_dim is positive. Keep the existing embedding and vector_dim skip behavior
inside the loop, and use the constant vector_dim for the single index
initialization.
nodes/src/nodes/vectordb_elasticsearch/requirements.txt (1)

1-3: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Pin requests to a safe minimum version.

Based on learnings, please pin requests to a safe minimum version (e.g., requests>=2.34.2) in the node's requirements.txt to avoid resolving vulnerable older versions and to keep dependency scans accurate.

📦 Proposed fix
 elasticsearch>=8.0.0,<9.0.0
 opensearch-py==3.2.0
 numpy
+requests>=2.34.2
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

elasticsearch>=8.0.0,<9.0.0
opensearch-py==3.2.0
numpy
requests>=2.34.2
🧰 Tools
🪛 OSV Scanner (2.4.0)

[MEDIUM] 1-1: idna 3.9.0: undefined

(PYSEC-2026-215)


[MEDIUM] 1-1: idna 3.9.0: Internationalized Domain Names in Applications (IDNA): Specially crafted inputs to idna.encode() can bypass CVE-2024-3651 fix

(GHSA-65pc-fj4g-8rjx)


[HIGH] 1-1: urllib3 1.26.20: undefined

(PYSEC-2026-141)


[HIGH] 1-1: urllib3 1.26.20: urllib3 streaming API improperly handles highly compressed data

(PYSEC-2026-1994)


[HIGH] 1-1: urllib3 1.26.20: Decompression-bomb safeguards bypassed when following HTTP redirects (streaming API)

(PYSEC-2026-1996)


[HIGH] 1-1: urllib3 1.26.20: urllib3 allows an unbounded number of links in the decompression chain

(PYSEC-2026-1998)


[HIGH] 1-1: urllib3 1.26.20: urllib3 redirects are not disabled when retries are disabled on PoolManager instantiation

(PYSEC-2026-1999)


[HIGH] 1-1: urllib3 1.26.20: urllib3 streaming API improperly handles highly compressed data

(GHSA-2xpw-w6gg-jr37)


[HIGH] 1-1: urllib3 1.26.20: Decompression-bomb safeguards bypassed when following HTTP redirects (streaming API)

(GHSA-38jv-5279-wg99)


[HIGH] 1-1: urllib3 1.26.20: urllib3 allows an unbounded number of links in the decompression chain

(GHSA-gm62-xv2j-4w53)


[HIGH] 1-1: urllib3 1.26.20: urllib3 redirects are not disabled when retries are disabled on PoolManager instantiation

(GHSA-pq67-6m6q-mj2v)


[HIGH] 1-1: urllib3 1.26.20: urllib3: Sensitive headers forwarded across origins in proxied low-level redirects

(GHSA-qccp-gfcp-xxvc)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_elasticsearch/requirements.txt` around lines 1 - 3,
Update the node dependency list in requirements.txt by adding requests with a
safe minimum constraint of requests>=2.34.2, while preserving the existing
elasticsearch, opensearch-py, and numpy dependencies.

Source: Learnings

nodes/src/nodes/vectordb_milvus/IGlobal.py (1)

65-90: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n## IGlobal.py slice\n'
sed -n '50,110p' nodes/src/nodes/vectordb_milvus/IGlobal.py

printf '\n## milvus.py connection usages\n'
rg -n "connections\.connect|uri=|https://|http://" nodes/src/nodes/vectordb_milvus -n

printf '\n## milvus.py relevant slice\n'
sed -n '1,220p' nodes/src/nodes/vectordb_milvus/milvus.py

Repository: rocketride-org/rocketride-server

Length of output: 13616


🏁 Script executed:

#!/bin/bash
set -euo pipefail
nl -ba nodes/src/nodes/vectordb_milvus/IGlobal.py | sed -n '1,130p'

Repository: rocketride-org/rocketride-server

Length of output: 210


Cloud validation probe needs the same https:// URI as runtime — uri = f'{host_raw}:{port}' omits the scheme that Store.__init__ uses for cloud profiles, so token-based configs can fail validation with a false warning. Use the same cloud URI shape as the runtime path.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_milvus/IGlobal.py` around lines 65 - 90, Update the
token-based cloud branch in the validation connection flow to construct the URI
with the same https:// scheme and host/port shape used by Store.__init__. Keep
host_raw and port unchanged for local connections, and continue passing the
resulting URI to connections.connect.
nodes/src/nodes/vectordb_milvus/milvus.py (2)

470-478: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Silently swallowed delete failure risks duplicate/stale chunks on document replace.

Unlike remove() (lines 538-543), which re-raises after logging a client.delete failure, addChunks() only logs via engLib.debug and continues to insert the new chunks regardless of whether the old ones for the same objectId were actually removed. If the delete fails (timeout, transient error, etc.), old and new chunks for the same objectId will coexist, silently corrupting search results and render() output (duplicated/interleaved content) with no signal to the caller.

🐛 Proposed fix
             try:
                 # Delete entities
                 self.client.delete(collection_name=self.collection, filter=filter_condition)
             except Exception as e:
                 engLib.debug(f'Error deleting old chunks: {e}')
+                raise
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        # Erase all documents/chunks associated with that ObjectId in one operation
        # so we can cleanly insert the new version
        if len(objectIds.keys()):
            filter_condition = f"meta['objectId'] in [{', '.join(json.dumps(k) for k in objectIds.keys())}]"
            try:
                # Delete entities
                self.client.delete(collection_name=self.collection, filter=filter_condition)
            except Exception as e:
                engLib.debug(f'Error deleting old chunks: {e}')
                raise
🧰 Tools
🪛 ast-grep (0.44.1)

[info] 472-472: use jsonify instead of json.dumps for JSON output
Context: json.dumps(k)
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_milvus/milvus.py` around lines 470 - 478, Update the
delete error handling in addChunks so failures from self.client.delete are
logged and then re-raised, matching the behavior in remove(). Do not continue
inserting replacement chunks when deletion of existing objectId chunks fails;
preserve the current successful deletion and insertion flow.

663-674: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

Critical: render() reads a non-existent top-level chunkId field — will raise KeyError on every non-empty result.

The query only requests output_fields=['meta', 'content'], and the collection schema (see _createCollection, lines 150-169) has no top-level chunkId field — chunkId lives inside the JSON meta blob. getPaths() correctly accesses it via record.get('meta', {}).get('chunkId'), but here chunk = point['chunkId'] will raise KeyError for every returned point, breaking the entire "Rendering" feature described in the README whenever an object has stored chunks.

Additionally, unlike every sibling store's render() (chroma, qdrant, pinecone, elasticsearch — per the provided graph context), this loop is missing the upper-bound guard (if chunk >= offset + self.renderChunkSize: continue), so once the KeyError is fixed, an out-of-range chunkId from a stale/dirty query result could still cause an IndexError on text[index] = content.

🐛 Proposed fix
             for point in results:
                 content = point['content']
-                chunk = point['chunkId']
+                meta = point.get('meta', {}) or {}
+                chunk = meta.get('chunkId')
                 index = chunk - offset
 
                 # Should never happen since we gave it an offset
                 if chunk < offset:
                     continue
 
+                # Should never happen since we gave it a range
+                if chunk >= offset + self.renderChunkSize:
+                    continue
+
                 text[index] = content
                 if index > lastIndex:
                     lastIndex = index
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

            for point in results:
                content = point['content']
                meta = point.get('meta', {}) or {}
                chunk = meta.get('chunkId')
                index = chunk - offset

                # Should never happen since we gave it an offset
                if chunk < offset:
                    continue

                # Should never happen since we gave it a range
                if chunk >= offset + self.renderChunkSize:
                    continue

                text[index] = content
                if index > lastIndex:
                    lastIndex = index
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_milvus/milvus.py` around lines 663 - 674, Update the
result-processing loop in render() to read chunkId from each point’s meta JSON,
matching getPaths(), instead of accessing a non-existent top-level field. Add
the upper-bound guard so chunks at or beyond offset + self.renderChunkSize are
skipped before indexing text, while preserving the existing lower-bound check
and rendering behavior for valid chunks.
nodes/src/nodes/vectordb_pinecone/IGlobal.py (1)

177-182: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

pinecone python SDK list_indexes return type IndexList index model .get name spec

💡 Result:

In the Pinecone Python SDK, the list_indexes method returns an IndexList object [1][2][3]. This object is a wrapper around a collection of IndexModel instances and provides convenience methods for interacting with the returned data [4][5]. Key details regarding the return type and object structure include: Accessing Index Models: While the IndexList object is iterable and supports length checks (len) and direct indexing (e.g., indexes), it is a wrapper class [4][5]. To access the actual list of IndexModel objects, you should access the.indexes attribute (e.g., pc.list_indexes.indexes) [4][5][3]. Convenience Methods: -.names: Returns a list of strings containing the names of all indexes in the project [4][5]. -.to_dict: Returns a serializable dictionary representation of the index list, where each entry is converted to a dictionary [4][5]. IndexModel Structure: Each IndexModel instance (accessible via iteration or direct indexing on the IndexList) contains metadata about the index, including attributes such as: - name: The name of the index [6]. - spec: The specification of the index (e.g., serverless or pod-based configuration) [6][7]. - status: The current status of the index (e.g., ready state) [6][7]. - metric, dimension, etc.: Other relevant configuration details [6][7]. Example Usage: from pinecone import Pinecone pc = Pinecone(api_key='YOUR_API_KEY') indexes = pc.list_indexes # Iterate over index models for index in indexes.indexes: print(f"Name: {index.name}, Spec: {index.spec}") # Get list of names names = indexes.names # Access a specific model directly first_index = indexes.indexes

Citations:


🌐 Web query:

Pinecone Python SDK IndexModel .get() spec attribute list_indexes iterable direct indexing

💡 Result:

In the Pinecone Python SDK, IndexModel objects and the associated list_indexes response provide specific methods for accessing index details and iterating over collections. IndexModel and the spec attribute: The IndexModel class, which is returned by methods like pc.indexes.describe("index_name"), contains a spec attribute [1][2]. This attribute is an instance of IndexSpec, which holds deployment configuration (e.g., serverless, pod, or byoc) [1]. While you cannot use direct dictionary-style indexing (e.g., index_model['spec']) to access this, you can access it via dot notation (e.g., index_model.spec.serverless.cloud) [2]. To get a dictionary representation of the entire model, you can use the.to_dict method, which recursively converts the spec and other nested fields into a plain dictionary [1][3]. list_indexes and iteration: The pc.list_indexes method returns an IndexList object, which acts as a wrapper around a list of IndexModel instances [4]. This object is iterable, meaning you can loop over it directly to access each IndexModel [5][4]. For example: for index in pc.list_indexes: print(index.name) Additionally, the IndexList object provides a.names method if you only need the names of the indexes, which returns an iterable list of strings [5][4]. Direct Indexing: The SDK does not support direct indexing (dictionary-style lookup) on IndexModel or IndexList objects themselves [4]. Access must be performed through their defined attributes or helper methods like.to_dict [1][4]. To interact with a specific index's data plane, you use pc.index("index_name") to obtain an Index client instance [6][7].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file='nodes/src/nodes/vectordb_pinecone/IGlobal.py'
nl -ba "$file" | sed -n '160,195p'

printf '\n---\n'

rg -n "list_indexes|IndexModel|to_dict|spec.serverless|serverless" nodes/src/nodes/vectordb_pinecone -S

Repository: rocketride-org/rocketride-server

Length of output: 210


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file='nodes/src/nodes/vectordb_pinecone/IGlobal.py'

awk 'NR>=160 && NR<=195 { printf "%d:%s\n", NR, $0 }' "$file"

printf '\n---\n'

rg -n "list_indexes|existing_collection|get\\('name'\\)|get\\('spec'\\)|serverless" "$file" -n -S

Repository: rocketride-org/rocketride-server

Length of output: 2977


Use Pinecone model attributes here client.list_indexes() yields IndexModel items, so index.get('name') and existing_collection.get('spec', {}) won’t work; use index.name / existing_collection.spec (or to_dict() first) before checking mode compatibility.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_pinecone/IGlobal.py` around lines 177 - 182, Update
the collection lookup and mode validation in the Pinecone initialization flow to
use IndexModel attributes: compare `index.name` when finding
`existing_collection`, then inspect `existing_collection.spec` for serverless
mode instead of calling `.get()` on the model. Preserve the existing
compatibility-check behavior.

Source: Linters/SAST tools

nodes/src/nodes/vectordb_pinecone/pinecone.py (2)

238-245: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

searchKeyword double-converts the filter and passes a dict into get().

get() unconditionally calls self._convertFilter(docFilter=docFilter), which accesses DocFilter attributes (.isDeleted, .nodeId, …). Here filter is already a converted {'$and': [...]} dict, so get() will raise AttributeError on the dict, and the appended $contains predicate is discarded even if it didn't crash. Keyword search is broken on any invocation.

Pass the DocFilter through and inject the keyword predicate inside the query path, or add a raw-filter branch to get().

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_pinecone/pinecone.py` around lines 238 - 245, Update
searchKeyword so it does not pass the already converted filter dict to get(),
since get() unconditionally calls _convertFilter and expects a DocFilter.
Preserve the keyword predicate by either passing the original DocFilter and
applying the content condition within the query path, or adding an explicit
raw-filter path to get() that supports the converted filter.

288-290: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Score threshold is read from the wrong (misspelled) attribute, so it never applies.

__init__ sets self.threshold_search (Line 90), but this filter reads self.threshhold_search (the misspelled class attribute at Line 68, permanently 0.0). Every non-negative score passes, so the configured score threshold is silently ignored for semantic search.

🐛 Proposed fix
-        points_satisfy_threshold = [point for point in points if point['score'] >= self.threshhold_search]
+        points_satisfy_threshold = [point for point in points if point['score'] >= self.threshold_search]

Also rename the class attribute at Line 68 (threshhold_search → threshold_search) so the declared field matches the value set in __init__.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.


        # Have to get similarity score threshold post query
        points_satisfy_threshold = [point for point in points if point['score'] >= self.threshold_search]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_pinecone/pinecone.py` around lines 288 - 290, Fix
the threshold attribute naming in the Pinecone node: rename the class attribute
threshhold_search to threshold_search and update the points_satisfy_threshold
filter to read self.threshold_search, matching the value initialized by __init__
so semantic search applies the configured score threshold.
nodes/src/nodes/vectordb_pinecone/README.md (1)

54-59: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Default-profile documentation contradicts the schema.

This section marks serverless-dense as the default ("The default profile is serverless-dense"), but the Fields table (Line 36) and the generated schema (Line 112) both give the profile default as "pod-based". Reconcile so the docs state a single, correct default.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_pinecone/README.md` around lines 54 - 59, Reconcile
the Pinecone profile default documentation with the schema by updating the
conflicting default references in the README, including the profile table and
the statement below it, so they consistently identify pod-based as the default.
nodes/src/nodes/vectordb_qdrant/qdrant.py (1)

140-149: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

Consider closing the client in __del__.

__del__ clears attributes but never calls self.client.close(), unlike IGlobal.validateConfig() which explicitly closes its probe client. Since QdrantClient holds open HTTP connections, closing it here would be more consistent and avoid relying on GC to release sockets.

♻️ Proposed fix
     def __del__(self):
         """
         Deinitializes the qdrant client.
         """
         # Deinit everything we did
         self.apikey = None
         self.collection = ''
         self.renderChunkSize = 0
         self.similarity = 'Cosine'
+        if self.client is not None:
+            try:
+                self.client.close()
+            except Exception:
+                pass
         self.client = None
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def __del__(self):
        """
        Deinitializes the qdrant client.
        """
        # Deinit everything we did
        self.apikey = None
        self.collection = ''
        self.renderChunkSize = 0
        self.similarity = 'Cosine'
        if self.client is not None:
            try:
                self.client.close()
            except Exception:
                pass
        self.client = None
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_qdrant/qdrant.py` around lines 140 - 149, Update
Qdrant’s __del__ method to close the existing self.client before clearing it,
while safely handling the client being unset or unavailable. Preserve the
current attribute-reset behavior after cleanup.
nodes/src/nodes/vectordb_weaviate/IGlobal.py (1)

71-154: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

validateConfig infers cloud/local from the hostname instead of the configured mode, which can diverge from what Store.__init__ actually connects to.

Store.__init__ (weaviate.py) decides cloud vs. local via config.get('mode'), but validateConfig here re-derives the same decision from host string patterns (localhost/127.*). If a user picks the "local" profile with a non-loopback host (common with Docker/K8s service names or private IPs), or "cloud" pointed at a tunnel, validateConfig will probe the wrong protocol/endpoint (skipping the gRPC readiness check for a real local connection, or vice versa), giving misleading save-time validation results relative to actual runtime behavior.

🐛 Proposed fix
             config = Config.getNodeConfig(self.glb.logicalType, self.glb.connConfig)
             host = (config.get('host', '')).strip()
             port = config.get('port')
             grpc_port = config.get('grpc_port')
             apikey = (config.get('apikey', '')).strip()
             collection = (config.get('collection', '')).strip()
+            profile = config.get('mode')
@@
-            # Decide cloud vs local: treat localhost/127.* as local; otherwise cloud
-            is_local = host in ('localhost', '127.0.0.1') or host.startswith('127.')
-            is_cloud = not is_local
+            # Match the same decision Store.__init__ uses at runtime
+            is_cloud = profile == 'cloud'
+            is_local = not is_cloud
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def validateConfig(self):
        """
        Validate Weaviate config at save-time with a fast, SDK-driven probe.

        - Cloud: connect_to_weaviate_cloud + client.is_ready()
        - Local: connect_to_local (host, port, grpc_port) + client.is_ready()
        - Collection name format check (no existence check)
        - Minimal timeouts; SDK/HTTP exceptions surfaced without truncation
        """
        try:
            # Get config
            config = Config.getNodeConfig(self.glb.logicalType, self.glb.connConfig)
            host = (config.get('host', '')).strip()
            port = config.get('port')
            grpc_port = config.get('grpc_port')
            apikey = (config.get('apikey', '')).strip()
            collection = (config.get('collection', '')).strip()
            profile = config.get('mode')

            # Collection name validation (official Weaviate rule)
            if not WEAVIATE_COLLECTION_RE.fullmatch(collection or ''):
                warning(
                    'Start with uppercase; only letters/numbers/underscore; no spaces or special characters (/ , . : ; \' " { } ( ) % ^ & $ # @ - ! ? * + = [ ] | \\ ~ < >).'
                )
                return

            # Load dependencies before importing SDK
            from depends import depends  # type: ignore

            requirements = os.path.dirname(os.path.realpath(__file__)) + '/requirements.txt'
            depends(requirements)

            # Imports used for SDK clients
            import weaviate  # type: ignore
            from weaviate.classes.init import AdditionalConfig, Timeout  # type: ignore
            import httpx  # type: ignore
            import socket  # type: ignore

            # Match the same decision Store.__init__ uses at runtime
            is_cloud = profile == 'cloud'
            is_local = not is_cloud

            # Build base URL (add scheme if missing)
            if is_local:
                base_url = host if host.startswith('http') else f'http://{host}'
                if port:
                    base_url = f'{base_url}:{port}'
            else:
                base_url = host if host.startswith('https://') else f'https://{host}'

            if is_cloud:
                # Cloud: REST + API key only; do not probe gRPC
                url = base_url.rstrip('/') + '/v1/meta'
                headers = {'Authorization': f'Bearer {apikey}'} if apikey else {}
                r = httpx.get(url, headers=headers, timeout=3)
                r.raise_for_status()
                return
            else:
                # Local: use SDK client and a lightweight call (no API key)
                client = weaviate.connect_to_local(
                    host=host,
                    port=port,
                    grpc_port=grpc_port,
                    additional_config=AdditionalConfig(timeout=Timeout(init=3, query=3)),
                )
                try:
                    # REST check via SDK (hits /v1/schema)
                    client.collections.list_all()
                    # gRPC READY (grpc_port is required by UI)
                    gp = int(grpc_port)
                    try:
                        import grpc  # type: ignore

                        grpc.channel_ready_future(grpc.insecure_channel(f'{host}:{gp}')).result(timeout=3)
                    except ImportError:
                        with socket.create_connection((host, gp), timeout=3):
                            pass
                finally:
                    try:
                        client.close()
                    except Exception:
                        pass

        except Exception as e:
            warning(_format_error(e))
🧰 Tools
🪛 ast-grep (0.44.1)

[warning] 113-113: Do not make http calls without encryption
Context: f'http://{host}'
Note: [CWE-319] Cleartext Transmission of Sensitive Information.

(requests-http)


[warning] 123-123: Request-controlled URL passed to httpx; validate against an allowlist to prevent SSRF.
Context: httpx.get(url, headers=headers, timeout=3)
Note: [CWE-918] Server-Side Request Forgery (SSRF).

(avoid-ssrf)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_weaviate/IGlobal.py` around lines 71 - 154, Update
validateConfig to determine cloud versus local using the configured mode value,
matching the mode selection used by Store.__init__ in weaviate.py, instead of
inferring it from host patterns. Preserve the existing cloud HTTP probe and
local SDK/gRPC readiness checks, but route them according to the selected mode
so save-time validation matches runtime behavior.
nodes/src/nodes/vectordb_weaviate/weaviate.py (2)

489-541: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Per-object update loop for markDeleted/markActive — consider batching.

Both methods issue one collectionObj.data.update(...) call per matching point. For soft-delete/restore operations touching thousands of chunks (a plausible scenario per the object-level re-ingestion pattern documented in the README), this becomes an O(n) round-trip cost. Worth checking whether Weaviate's batch API (already used in addChunks) supports partial-property batch updates to reduce this to a single batched call.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_weaviate/weaviate.py` around lines 489 - 541, Update
markDeleted and markActive to use Weaviate’s batch API, consistent with
addChunks, instead of calling collectionObj.data.update once per point. Batch
the matching point UUIDs with the appropriate isDeleted value while preserving
the existing collection checks, empty-input handling, and filtering behavior.

543-606: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the relevant file sections and nearby query usages
python3 - <<'PY'
from pathlib import Path
path = Path('nodes/src/nodes/vectordb_weaviate/weaviate.py')
lines = path.read_text().splitlines()
for start, end in [(300, 420), (543, 606)]:
    print(f"\n--- {path} lines {start}-{end} ---")
    for i in range(start, min(end, len(lines)) + 1):
        print(f"{i:4d}: {lines[i-1]}")
PY

# Search for fetch_objects call sites in this file
rg -n "fetch_objects\(" nodes/src/nodes/vectordb_weaviate/weaviate.py

Repository: rocketride-org/rocketride-server

Length of output: 8445


render() needs an explicit limit on fetch_objects

Without limit=self.renderChunkSize, Weaviate falls back to its default page size (10), so any document with more than 10 chunks in a window is rendered incompletely and the remaining chunks are dropped.

🐛 Proposed fix
             points = self.collectionObj.query.fetch_objects(
                 filters=(
                     Filter.by_property('objectId').equal(objectId)
                     & Filter.by_property('chunkId').greater_or_equal(offset)
                     & Filter.by_property('chunkId').less_than(offset + self.renderChunkSize)
-                )
+                ),
+                limit=self.renderChunkSize,
             )
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def render(self, objectId: str, callback: Callable[[str], None]) -> None:
        """
        Given an object id, render the complete document.

        Rehydrates all the chunks into the proper order.
        """
        # By definition, if the collection does not exists, there
        # is nothing to render
        if not self.doesCollectionExist():
            return

        # Since chunks are returned in any order, and a single objectId
        # may contain tens of thousands of chunks, we grave them one
        # group at a time (renderChunkSize), put them into an array,
        # join them and call the callback
        offset = 0
        while True:
            # Build filter for getting a set of chunks within the offset range
            points = self.collectionObj.query.fetch_objects(
                filters=(
                    Filter.by_property('objectId').equal(objectId)
                    & Filter.by_property('chunkId').greater_or_equal(offset)
                    & Filter.by_property('chunkId').less_than(offset + self.renderChunkSize)
                ),
                limit=self.renderChunkSize,
            )

            # Create a renderChunkSize array with empty
            # entries. This will allow us to join even when
            # a chunk doesn't come back
            # Process the results
            text = [''] * self.renderChunkSize
            lastIndex = -1

            for point in points.objects:
                content = point.properties.get('content')
                chunk = point.properties.get('chunkId')
                index = chunk - offset

                # Should never happen since we gave it an offset
                if chunk < offset:
                    continue

                text[index] = content
                if index > lastIndex:
                    lastIndex = index

            # Compute the number of items we are going to process
            numberOfItems = lastIndex + 1

            # If we got no items back, we are done
            if numberOfItems < 1:
                break

            # Join it together
            fullText = ''.join(text[0:numberOfItems])

            # Call the output function
            callback(fullText)

            # If we got less than we asked for, must be done
            if numberOfItems < self.renderChunkSize:
                break

            offset += self.renderChunkSize
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/vectordb_weaviate/weaviate.py` around lines 543 - 606, Update
the fetch_objects call inside render to pass limit=self.renderChunkSize,
ensuring each chunk window requests the full configured batch rather than
Weaviate’s default page size. Preserve the existing offset filtering, ordering,
callback behavior, and termination logic.
nodes/test/vectordb_astra/test_convert_filter.py (1)

15-15: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Stale import path: nodes.astra_db.astra_db no longer exists after the rename.

This PR relocates the module to nodes/src/nodes/vectordb_astra/, but this test still imports Store from nodes.astra_db.astra_db. That package path should no longer resolve once the folder is renamed, so this test will fail with ModuleNotFoundError/ImportError rather than exercising _convertFilter. This also contradicts the AI summary's claim that test import paths were updated for this cohort.

🐛 Proposed fix
-from nodes.astra_db.astra_db import Store
+from nodes.vectordb_astra.astra_db import Store
#!/bin/bash
# Confirm the module was moved and no `nodes.astra_db` package remains, and check other astra tests for the correct import path.
fd -a astra_db.py
rg -n 'nodes\.astra_db|nodes\.vectordb_astra' nodes/test -g '*.py'
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/test/vectordb_astra/test_convert_filter.py` at line 15, Update the
Store import in test_convert_filter.py to use the relocated nodes.vectordb_astra
module path, matching the package structure used by the moved implementation.
Remove the stale nodes.astra_db.astra_db reference so the test can import Store
and exercise _convertFilter.
nodes/test/vectordb_milvus/test_addchunks_return.py (1)

19-23: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Stale nodes.milvus.milvus import inside the skipped test.

This file was moved under nodes/test/vectordb_milvus/, but the import inside the test body still targets the pre-rename module path. Since the test is skipped, this won't fail CI now, but it will break with ModuleNotFoundError as soon as the skip is removed.

🐛 Proposed fix
-    from nodes.milvus.milvus import Store
+    from nodes.vectordb_milvus.milvus import Store
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`@pytest.mark.skip`(reason='todo: refactor this unit test')
def test_milvus_addchunks_returns_none_when_create_collection_fails():
    """``addChunks`` must return None when ``createCollection`` fails."""
    from nodes.vectordb_milvus.milvus import Store
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/test/vectordb_milvus/test_addchunks_return.py` around lines 19 - 23,
Update the import inside
test_milvus_addchunks_returns_none_when_create_collection_fails to reference the
current Milvus module path after the package move, removing the stale
nodes.milvus.milvus target so the test can run when its skip is removed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Renames the vector-store node family to the vectordb_* prefix and the graph-first database nodes to graph_*, updating service registrations, documentation redirects, and tests to match the new module paths while keeping runtime resolution stable via unchanged protocol/prefix.

Changes:

  • Updated node service registrations (path, and for graph nodes also classType) to the new vectordb_* / graph_* module names.
  • Renamed/added node docs and dependency manifests under the new node directories.
  • Updated tests and docs redirects so imports and doc URLs continue to resolve after the renames.

Reviewed changes

Copilot reviewed 32 out of 115 changed files in this pull request and generated no comments.

Show a summary per file
File Description
packages/docs/redirects.ts Adds redirects from old /nodes/<old> doc routes to the new vectordb_* / graph_* routes.
nodes/test/weaviate/test_record_updates.py Updates direct module-path loading to vectordb_weaviate.
nodes/test/tool_xtrace_memory/test_tools.py Scopes engine-dependency stubs to import-time and restores sys.modules afterward.
nodes/test/tool_mem0/test_tools.py Scopes engine-dependency stubs to import-time and restores sys.modules afterward.
nodes/test/tool_filesystem/test_read_size_cap.py Scopes rocketlib stub to import-time and restores it afterward.
nodes/test/test_db_hydradb.py Updates node directory reference to graph_hydradb.
nodes/test/test_db_arango.py Updates node directory reference to graph_arango.
nodes/test/test_currency_convert_explicit.py Scopes rocketlib stub to import-time and restores it afterward.
nodes/test/test_anomaly_detector.py Scopes rocketlib stub to import-time and restores it afterward.
nodes/test/qdrant/test_search_threshold.py Updates direct module load path to vectordb_qdrant.
nodes/test/qdrant/test_mark_deleted.py Updates direct module load path to vectordb_qdrant.
nodes/test/pinecone/test_record_updates.py Updates direct module load path to vectordb_pinecone.
nodes/test/ocr/test_model_server_ocr.py Updates intra-test references to renamed test modules.
nodes/test/milvus/test_search_semantic.py Updates sys.path insertion to vectordb_milvus.
nodes/test/milvus/test_filter_injection.py Updates sys.path insertion to vectordb_milvus.
nodes/test/guardrails/test_all.py Scopes rocketlib stub to import-time and restores it afterward.
nodes/test/conftest.py Updates skip-list entry from index_search to vectordb_elasticsearch.
nodes/test/cloud_tts/test_cloud_tts.py Scopes core stubs to import-time and restores sys.modules afterward.
nodes/test/chroma/test_convert_filter_explicit_none.py Updates direct module load path to vectordb_chroma.
nodes/test/chroma/test_coerce_port.py Updates direct module load path to vectordb_chroma.
nodes/test/atlas/test_isdeleted_filter.py Updates direct module load path to vectordb_atlas and scopes pydantic stub.
nodes/src/nodes/vectordb_weaviate/services.json Updates Python module path to nodes.vectordb_weaviate.
nodes/src/nodes/vectordb_weaviate/requirements.txt Adds node dependency list under the renamed node directory.
nodes/src/nodes/vectordb_weaviate/README.md Adds co-located node documentation under the renamed node directory.
nodes/src/nodes/vectordb_weaviate/IInstance.py Adds node instance transform under the renamed node directory.
nodes/src/nodes/vectordb_weaviate/IGlobal.py Adds node global transform + save-time config validation under the renamed node directory.
nodes/src/nodes/vectordb_weaviate/IEndpoint.py Adds endpoint transform under the renamed node directory.
nodes/src/nodes/vectordb_weaviate/init.py Adds node module exports under the renamed node directory.
nodes/src/nodes/vectordb_qdrant/services.json Updates Python module path to nodes.vectordb_qdrant.
nodes/src/nodes/vectordb_qdrant/requirements.txt Adds node dependency list under the renamed node directory.
nodes/src/nodes/vectordb_qdrant/README.md Adds co-located node documentation under the renamed node directory.
nodes/src/nodes/vectordb_qdrant/qdrant.svg Adds node icon under the renamed node directory.
nodes/src/nodes/vectordb_qdrant/IInstance.py Adds node instance transform under the renamed node directory.
nodes/src/nodes/vectordb_qdrant/IGlobal.py Adds node global transform + save-time config validation under the renamed node directory.
nodes/src/nodes/vectordb_qdrant/IEndpoint.py Adds endpoint transform under the renamed node directory.
nodes/src/nodes/vectordb_qdrant/init.py Adds node module exports under the renamed node directory.
nodes/src/nodes/vectordb_pinecone/services.json Updates Python module path to nodes.vectordb_pinecone.
nodes/src/nodes/vectordb_pinecone/requirements.txt Adds node dependency list under the renamed node directory.
nodes/src/nodes/vectordb_pinecone/README.md Adds co-located node documentation under the renamed node directory.
nodes/src/nodes/vectordb_pinecone/pinecone.svg Adds node icon under the renamed node directory.
nodes/src/nodes/vectordb_pinecone/IInstance.py Adds node instance transform under the renamed node directory.
nodes/src/nodes/vectordb_pinecone/IGlobal.py Adds node global transform + save-time config validation under the renamed node directory.
nodes/src/nodes/vectordb_pinecone/IEndpoint.py Adds endpoint transform under the renamed node directory.
nodes/src/nodes/vectordb_pinecone/init.py Adds node module exports under the renamed node directory.
nodes/src/nodes/vectordb_milvus/services.json Updates Python module path to nodes.vectordb_milvus.
nodes/src/nodes/vectordb_milvus/requirements.txt Adds node dependency list under the renamed node directory.
nodes/src/nodes/vectordb_milvus/README.md Adds co-located node documentation under the renamed node directory.
nodes/src/nodes/vectordb_milvus/IInstance.py Adds node instance transform under the renamed node directory.
nodes/src/nodes/vectordb_milvus/IGlobal.py Adds node global transform + save-time config validation under the renamed node directory.
nodes/src/nodes/vectordb_milvus/IEndpoint.py Adds endpoint transform under the renamed node directory.
nodes/src/nodes/vectordb_milvus/init.py Adds node module exports under the renamed node directory.
nodes/src/nodes/vectordb_elasticsearch/services.opensearch.json Updates Python module path to nodes.vectordb_elasticsearch for OpenSearch service registration.
nodes/src/nodes/vectordb_elasticsearch/services.elasticsearch.json Updates Python module path to nodes.vectordb_elasticsearch for Elasticsearch service registration.
nodes/src/nodes/vectordb_elasticsearch/requirements.txt Adds combined Elasticsearch/OpenSearch dependency list under the renamed node directory.
nodes/src/nodes/vectordb_elasticsearch/opensearch/README.md Adds OpenSearch docs page under the renamed node directory.
nodes/src/nodes/vectordb_elasticsearch/opensearch.svg Adds OpenSearch icon under the renamed node directory.
nodes/src/nodes/vectordb_elasticsearch/IEndpoint.py Adds endpoint transform under the renamed node directory.
nodes/src/nodes/vectordb_elasticsearch/elasticsearch/README.md Adds Elasticsearch docs page under the renamed node directory.
nodes/src/nodes/vectordb_elasticsearch/elasticsearch.svg Adds Elasticsearch icon under the renamed node directory.
nodes/src/nodes/vectordb_elasticsearch/constants.py Introduces shared constants for the combined Elasticsearch/OpenSearch node.
nodes/src/nodes/vectordb_elasticsearch/init.py Adds node module exports under the renamed node directory.
nodes/src/nodes/vectordb_chroma/services.json Updates Python module path to nodes.vectordb_chroma.
nodes/src/nodes/vectordb_chroma/requirements.txt Adds node dependency list under the renamed node directory.
nodes/src/nodes/vectordb_chroma/README.md Adds co-located node documentation under the renamed node directory.
nodes/src/nodes/vectordb_chroma/IInstance.py Adds node instance transform under the renamed node directory.
nodes/src/nodes/vectordb_chroma/IGlobal.py Adds node global transform under the renamed node directory.
nodes/src/nodes/vectordb_chroma/IEndpoint.py Adds endpoint transform under the renamed node directory.
nodes/src/nodes/vectordb_chroma/chroma.svg Adds node icon under the renamed node directory.
nodes/src/nodes/vectordb_chroma/init.py Adds node module exports under the renamed node directory.
nodes/src/nodes/vectordb_atlas/services.json Updates Python module path to nodes.vectordb_atlas.
nodes/src/nodes/vectordb_atlas/requirements.txt Adds node dependency list under the renamed node directory.
nodes/src/nodes/vectordb_atlas/README.md Adds co-located node documentation under the renamed node directory.
nodes/src/nodes/vectordb_atlas/mongodb.svg Adds node icon under the renamed node directory.
nodes/src/nodes/vectordb_atlas/IInstance.py Adds node instance transform under the renamed node directory.
nodes/src/nodes/vectordb_atlas/IGlobal.py Adds node global transform + save-time config validation under the renamed node directory.
nodes/src/nodes/vectordb_atlas/IEndpoint.py Adds endpoint transform under the renamed node directory.
nodes/src/nodes/vectordb_atlas/init.py Adds node module exports under the renamed node directory.
nodes/src/nodes/vectordb_astra/services.json Updates Python module path to nodes.vectordb_astra.
nodes/src/nodes/vectordb_astra/requirements.txt Adds node dependency list under the renamed node directory.
nodes/src/nodes/vectordb_astra/README.md Adds co-located node documentation under the renamed node directory.
nodes/src/nodes/vectordb_astra/IInstance.py Adds node instance transform under the renamed node directory.
nodes/src/nodes/vectordb_astra/IGlobal.py Adds node global transform under the renamed node directory.
nodes/src/nodes/vectordb_astra/IEndpoint.py Adds endpoint transform under the renamed node directory.
nodes/src/nodes/vectordb_astra/astra_db.svg Adds node icon under the renamed node directory.
nodes/src/nodes/vectordb_astra/init.py Adds node module exports under the renamed node directory.
nodes/src/nodes/graph_hydradb/services.json Adds graph to classType and updates Python module path to nodes.graph_hydradb.
nodes/src/nodes/graph_hydradb/requirements.txt Adds node dependency list under the renamed node directory.
nodes/src/nodes/graph_hydradb/README.md Adds co-located node documentation under the renamed node directory.
nodes/src/nodes/graph_hydradb/IInstance.py Adds tool-based HydraDB instance implementation under the renamed node directory.
nodes/src/nodes/graph_hydradb/IGlobal.py Adds HydraDB global config/client wiring under the renamed node directory.
nodes/src/nodes/graph_hydradb/hydradb.svg Adds node icon under the renamed node directory.
nodes/src/nodes/graph_hydradb/hydradb_client.py Adds a thin HydraDB v2 REST client wrapper for the node tools.
nodes/src/nodes/graph_hydradb/init.py Bootstraps runtime dependency loading and exports node classes.
nodes/src/nodes/graph_arango/utils.py Adds shared utilities under the renamed node directory.
nodes/src/nodes/graph_arango/services.json Adds graph to classType and updates Python module path to nodes.graph_arango.
nodes/src/nodes/graph_arango/requirements.txt Adds node dependency list under the renamed node directory.
nodes/src/nodes/graph_arango/README.md Adds co-located node documentation under the renamed node directory.
nodes/src/nodes/graph_arango/arango.svg Adds node icon under the renamed node directory.
nodes/src/nodes/graph_arango/init.py Bootstraps driver installation and exports node classes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@dsapandora
dsapandora force-pushed the refactor/node-family-renames branch from 3de65d5 to 29a3dc3 Compare July 21, 2026 08:01
Copilot AI review requested due to automatic review settings July 21, 2026 08:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 34 out of 117 changed files in this pull request and generated no new comments.

@dsapandora
dsapandora force-pushed the refactor/node-family-renames branch from 29a3dc3 to 41a3274 Compare July 21, 2026 08:26
@dsapandora

Copy link
Copy Markdown
Collaborator Author

Heads up on CI here: this PR is renames-only and depends on #1642 (test isolation, fixes #1640).

CI on this branch stays red until #1642 lands on develop. The folder renames reorder pytest collection, which surfaces pre-existing sys.modules leaks in unrelated node tests (rocketlib/pydantic stubs never restored) — that's what #1642 fixes.

Validated end-to-end: applying #1642's fixes on top of this branch turned all three OS builds green (Ubuntu/macOS/Windows). Merge order: #1642 → develop, then rebase this.

@dsapandora
dsapandora force-pushed the refactor/node-family-renames branch from 41a3274 to bfa71f8 Compare July 22, 2026 15:51
Copilot AI review requested due to automatic review settings July 22, 2026 15:51
@dsapandora dsapandora changed the title refactor(nodes): rename vector db to vectordb_* and graph-first db to graph_* refactor(nodes): rename vector db to store_* and graph-first db to graph_* Jul 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 32 out of 120 changed files in this pull request and generated 5 comments.

Comment thread packages/docs/redirects.ts Resolved
Comment thread packages/docs/redirects.ts Resolved
Comment thread nodes/test/ocr/test_model_server_ocr.py Resolved
Comment thread nodes/test/ocr/test_model_server_ocr.py Resolved
Comment thread docs/README-nodes.md Resolved
…o graph_*

Rename the nine vector-store nodes to the store_ prefix (matching their `store`
classType) and the two graph-first databases (ArangoDB, HydraDB) to graph_.
Update each services.json path to the new python module and add graph to the
ArangoDB/HydraDB classType (database and tool kept). protocol and prefix are left
untouched, so saved pipes keep resolving; no services-version bump.

Docs redirects map the old /nodes/<old> URLs to the new ones. Test module imports
and path references follow the renamed src folders. constraints.lock and docs
static pages are regenerated by CI.

Node test folders keep their original names so pytest collection order is unchanged.
Copilot AI review requested due to automatic review settings July 22, 2026 20:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 30 out of 118 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

packages/docs/redirects.ts:120

  • PR description says the vector-store nodes are being renamed to the “vectordb_” prefix, but the actual redirects (and node folders/paths in this PR) use the “store_” prefix (e.g. /nodes/store_astra). Please update the PR description to match the implemented naming so readers aren’t misled about the intended convention.

@dsapandora
dsapandora marked this pull request as ready for review July 22, 2026 21:27

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 51

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@nodes/src/nodes/graph_arango/IGlobal.py`:
- Around line 184-204: Update _run_query_raw so affected_rows reflects writes
even when the query returns rows: use len(rows) as the affected count when rows
are present, while retaining the cursor-derived affected count when no rows are
returned. Keep the existing row collection and max_execute_rows enforcement
unchanged.

In `@nodes/src/nodes/graph_arango/IInstance.py`:
- Around line 175-224: Update the get_aql tool’s input_schema to document the
supported limit parameter alongside question, including its expected numeric
type and purpose, while preserving the existing _clamp_limit(args.get('limit'))
behavior.

In `@nodes/src/nodes/graph_arango/README.md`:
- Line 1: Update the README H1 and its “View source” link to use the renamed
graph_arango node and path, replacing the stale db_arango references while
preserving the existing link structure.

In `@nodes/src/nodes/store_astra/astra_db.py`:
- Around line 490-529: Update Store.render to process object chunks in batches
using the store’s renderChunkSize configuration, rather than collecting,
sorting, joining, and callbacking the entire object at once. Preserve chunk
ordering by fetching each batch in chunkId order, invoke callback once per
batch, and add or reuse the appropriate renderChunkSize attribute consistent
with sibling stores.
- Around line 318-357: Update getPaths to exclude soft-deleted records by adding
the standard meta.isDeleted: False condition to its aggregation match filter,
while preserving the existing parent filter, pagination, and path-building
behavior.
- Around line 276-299: Update searchSemantic to raise a clear exception when
query.embedding is None and reject any non-zero docFilter.offset before querying
Astra. Preserve the existing collection check and semantic search flow for valid
embeddings with offset zero.
- Around line 147-200: Escape the user-supplied parent path before assigning it
to the $regex filter in _convertFilter. Use the standard regex-escaping utility
so metacharacters in docFilter.parent are matched literally, while preserving
the existing parent filter behavior and all other filter clauses.
- Around line 358-435: Update the near-zero-vector validation inside
addChunks.flush so the schema control document created by the zero-embedding
workaround is retained for insertion. Ensure the validity check or
schema-specific handling accepts that boosted control vector while continuing to
filter genuinely invalid vectors for other documents, preserving
doesCollectionExist control-document detection.
- Around line 62-90: Update api_endpoint initialization in the store setup to
require a configured value, normalize it by trimming whitespace, removing the
http/https scheme and surrounding trailing slashes as described by the existing
comment, and validate the resulting endpoint before passing it to get_database.
Raise a clear configuration error for missing or invalid endpoints, while
preserving the existing collection_name, similarity, and application_token
handling.

In `@nodes/src/nodes/store_astra/README.md`:
- Around line 84-99: Regenerate the documentation for the renamed store_astra
node using the nodes:docs-generate workflow so the generated Source link points
to nodes/src/nodes/store_astra instead of the obsolete astra_db path. Do not
manually edit the generated block.

In `@nodes/src/nodes/store_atlas/atlas.py`:
- Around line 351-352: Replace the shallow sys.getsizeof(doc) calculation in the
document batching loop with an estimate of the serialized payload that includes
content bytes and embedding element storage, or BSON-encode the document for an
exact measure. Keep accumulating the estimate in sum_size so the existing
payload_limit flush condition reflects actual document size.
- Around line 318-353: Update addChunks to validate that every chunk has a
non-None embedding before computing and deleting existing object IDs. Move the
embedding validation ahead of the objectIds deletion block, while preserving the
existing exception behavior and document preparation flow.
- Around line 270-276: Initialize results to a safe empty collection before the
try block, or return an appropriate empty-document result directly from the
OperationFailure handler, so _convertToDocs receives a defined value when
aggregate fails. Preserve the existing successful aggregation and logging
behavior.
- Around line 236-249: Remove the redundant model-parameter collection check and
its misleading validation comment from searchSemantic. Keep the initial
doesCollectionExist() guard returning [] for missing collections, while
preserving the separate embedding and offset validations.
- Around line 294-316: Escape externally supplied parent filter values before
constructing MongoDB `$regex` conditions in both `getPaths()` and
`_convertFilter()`. Apply `re.escape(...)` to `parent` and `docFilter.parent`
while preserving the existing case-insensitive matching behavior and filter
structure.

In `@nodes/src/nodes/store_atlas/README.md`:
- Around line 74-77: Regenerate the documentation for the renamed store_atlas
node using nodes:docs-generate so the generated “View source” link targets
nodes/src/nodes/store_atlas. Do not manually edit content within the
ROCKETRIDE:GENERATED:PARAMS block.

In `@nodes/src/nodes/store_chroma/chroma.py`:
- Around line 245-253: Update the cosine branch in the score calculation around
self.similarity to convert Chroma’s distance into a similarity score using 1 -
distance / 2, so lower distances produce higher scores. Apply
self.threshold_search to this normalized score instead of the hardcoded 0.20
threshold, while preserving the existing non-cosine scoring path.
- Around line 498-503: The soft-delete and reactivation flows in markDeleted and
markActive incorrectly iterate Chroma result keys and call update with
unsupported arguments. At both affected sites in
nodes/src/nodes/store_chroma/chroma.py (lines 498-503 and 522-527), build
parallel ids and metadatas lists from the returned result columns, update each
metadata object’s isDeleted value while preserving all existing keys, then call
self.collectionObj.update with ids and metadatas.
- Around line 374-378: Update the getPaths filter construction to use the
chunkId metadata key with an equality condition for 0. When parent is provided,
combine the chunkId and parent conditions under a $and wrapper, preserving the
existing parent serialization.
- Around line 567-591: Update the collection.get call in render to request both
documents and metadatas, then read each row’s chunkId from the returned metadata
dictionary before applying the offset and range checks. Preserve the existing
text-array population logic and avoid treating metadata as an object attribute.

In `@nodes/src/nodes/store_elasticsearch/elasticsearch_store.py`:
- Around line 583-591: The payload batching logic in the bulk document loop must
measure serialized document content rather than the shallow dict size from
sys.getsizeof(doc). Update the size calculation near _run_bulk to use the
serialized _source (or equivalent payload-size estimate), and reuse
DEFAULT_TEXT_BATCH_SIZE instead of the hardcoded 500 when appropriate; preserve
flushing when either count or payload limits are reached.
- Around line 382-383: Update the offset validation in the Elasticsearch store’s
semantic-search flow to raise Exception instead of BaseException, preserving the
existing error message and condition for non-zero docFilter.offset values.
- Around line 93-116: Update the Elasticsearch configuration mapping before
Store.__init__ so deployment mode is resolved from the Elasticsearch
deployment/profile field, not config.mode. Pass the resolved profile-based
scheme or mode into Store, and ensure Store.__init__ uses that deployment value
when selecting the default URL scheme while preserving explicit http/https host
schemes.

In `@nodes/src/nodes/store_elasticsearch/README.md`:
- Line 187: Update the source metadata used by the README generator for the
Elasticsearch node so its generated source link targets
nodes/src/nodes/store_elasticsearch instead of nodes/src/nodes/index_search.
Then rerun nodes:docs-generate to regenerate the README block; do not edit the
generated link manually.
- Around line 37-75: Replace the outdated Elasticsearch configuration table with
entries matching the services.elasticsearch.json schema and Elasticsearch
loader: use the profile/vector fields shown in the lower table, remove
OpenSearch-only keys such as host, enabled, username, password, and collection,
and set provider to "elasticsearch" where applicable. Keep the documented
defaults and profile-specific fields accurate.
- Around line 124-126: Update the OpenSearch “Basic auth” documentation to
describe verify_certs=False and ssl_show_warn=False as an insecure compatibility
fallback, not suitable for production. Explicitly instruct production users with
self-signed certificates to configure CA certificate verification before
deployment, while preserving the existing authentication requirements.

In `@nodes/src/nodes/store_milvus/IGlobal.py`:
- Around line 74-77: Update the cloud validation branch in IGlobal.py to
construct the URI with the same https:// scheme used by the runtime Milvus
client before calling connections.connect. Preserve the raw host and port values
while ensuring token-based cloud validation receives a valid
http[s]://host[:port] URI.

In `@nodes/src/nodes/store_milvus/milvus.py`:
- Around line 417-431: Replace the json.dumps-based escaping in getPaths and the
delete filter within addChunks with the existing _escape_milvus_str helper.
Preserve the current filter semantics while routing both parent values through
the shared, tested escaping mechanism.
- Around line 450-520: In addChunks, validate that every chunk has a non-None
embedding before building objectIds or executing the existing deletion via
self.client.delete. Move the embedding check out of the insertion loop into a
preliminary validation pass, while preserving the current exception behavior and
upsert flow for valid batches.
- Around line 663-666: Update render() to read chunkId from the result’s meta
JSON/object, matching the existing getPaths() and _convertToDocs() handling,
instead of accessing point['chunkId'] directly. Preserve the current
chunk-offset calculation and rendering behavior after extracting the nested
value.
- Around line 104-109: Update the similarity validation in the Milvus
initialization logic to accept only COSINE, L2, and IP for the FLOAT_VECTOR
schema. Remove JACCARD, HAMMING, and BM25 from the allowed values while
preserving the existing exception for unsupported metrics.
- Around line 205-207: Update the docFilter.isTable condition to append a Milvus
filter using lowercase, unquoted boolean literals derived from the boolean
value, rather than _escape_milvus_str. Remove the extraneous tuple wrapper
around must_conditions.append so the condition is appended directly.
- Around line 144-193: The _createCollection method currently reverses the
expected success contract. Update its return values so an existing collection
and successful create_collection call return True, while exceptions during
creation return False; preserve the existing collection-existence check and
deadlock-avoidance behavior.
- Line 156: Remove the max_length argument from the INT64 primary-key field
definition in schema.add_field, while preserving the existing field_name,
datatype, and is_primary settings.

In `@nodes/src/nodes/store_pinecone/pinecone.py`:
- Line 68: Correct the misspelled class attribute threshhold_search to
threshold_search and update the searchSemantic filtering reference accordingly,
matching the self.threshold_search assignment in __init__. Ensure semantic
search uses the configured score threshold instead of the class-level default.
- Around line 226-249: Update searchKeyword and the related get flow so the
already-converted Pinecone filter is not passed back through _convertFilter as a
DocFilter. Separate filter conversion from filter application, or otherwise
reuse the converted filter directly, while preserving the existing DocFilter
conversion behavior for normal get callers.
- Around line 399-468: Update addChunks so flush only upserts pending points and
does not call self.remove or clear objectIds between batches. After the loop’s
final flush succeeds, delete the collected object IDs once, ensuring all new
chunks are written before stale data is removed and failures leave existing data
intact.

In `@nodes/src/nodes/store_pinecone/README.md`:
- Around line 52-59: Update the Profiles table and its concluding
default-profile statement in the README to use pod-based as the default,
matching the Fields table and generated Schema; mark serverless-dense as
non-default and pod-based as default without changing their deployment
descriptions.

In `@nodes/src/nodes/store_postgres/postgres.py`:
- Around line 408-478: Update the write methods _createCollection, addChunks,
remove, markDeleted, and markActive to wrap their cursor operations and commit
in try/except blocks; on any exception, call self.client.rollback() before
re-raising the original exception. Preserve existing behavior on successful
commits and early returns.
- Around line 424-439: The chunk insertion flow using self.client.cursor and
SQL_QUERIES['insert_chunk'] should batch all chunks into one database statement
instead of executing one INSERT per iteration. Build the complete row values and
shared column list first, then use psycopg2.extras.execute_values or an
equivalent multi-row INSERT, preserving vectorSize and modelName defaults before
committing once.
- Around line 123-127: Update the production connection in the store-postgres
constructor to pass psycopg2’s connect_timeout parameter, matching the 3-second
timeout used by IGlobal.validateConfig(). Keep the existing connection arguments
and register_vector(self.client) behavior unchanged.

In `@nodes/src/nodes/store_weaviate/IGlobal.py`:
- Around line 108-119: Update validateConfig’s cloud/local classification to
derive is_cloud from config.get('mode') == 'cloud', matching Store.__init__ in
weaviate.py, and derive is_local as its inverse. Remove the hostname-based
localhost/127.* heuristic while preserving the existing URL construction for
each resulting connection mode.

In `@nodes/src/nodes/store_weaviate/README.md`:
- Around line 126-129: Regenerate the documentation for the renamed
store_weaviate node using the nodes:docs-generate workflow so the generated
“View source” link targets nodes/src/nodes/store_weaviate instead of the
obsolete nodes/src/nodes/weaviate path. Do not edit the content between the
generated params markers manually.

In `@nodes/src/nodes/store_weaviate/requirements.txt`:
- Line 1: Update the authlib dependency declaration in requirements.txt to pin
it to version 1.7.2, matching the existing authlib==1.7.2 constraint in
constraints.lock.

In `@nodes/src/nodes/store_weaviate/weaviate.py`:
- Around line 415-469: Update addChunks to avoid deleting existing chunks before
the replacement batch has been imported successfully. Use an atomic or
rollback-safe replacement strategy—prefer UUID-based upsert when available,
otherwise insert the new chunks first and delete prior chunks only after the
batch completes without failed_objects—while preserving the existing objectId
replacement semantics.
- Around line 543-606: Update render() to replace the fixed-size text list
allocation with a sparse dictionary keyed by each point’s chunk - offset, so
memory usage scales with returned chunks rather than renderChunkSize. Preserve
chunk ordering, missing-chunk handling, lastIndex/numberOfItems calculation,
joined output, pagination, and callback behavior.
- Around line 489-514: Update markDeleted and markActive so their fetch_objects
queries do not rely on the default 10-result limit. Paginate through matching
objects in windows until no results remain, applying the existing isDeleted
update to every returned object. Preserve the collection-existence and
empty-objectIds early returns.
- Around line 267-277: Update the cosine branch in the score calculation around
self.similarity to normalize Weaviate distances using certainty semantics:
compute the score as 1 minus distance divided by 2. Keep the existing non-cosine
calculation, threshold check, and missing-distance behavior unchanged.
- Around line 302-326: The searchKeyword method currently interpolates the
entire QuestionText model into the Weaviate content filter. Update the filter
construction to use query.text so the wildcard pattern matches the question text
directly.

In `@nodes/test/conftest.py`:
- Around line 320-321: Track the temporary exclusion of store_elasticsearch for
re-enablement by adding a follow-up issue reference or TODO next to its entry in
the failing-test exclusion group. Keep the exclusion unchanged until dependency
`#1642` is resolved, and make the tracking explicit so the dynamically generated
tests can be restored afterward.

In `@packages/docs/redirects.ts`:
- Around line 183-197: Update the redirect entries in the redirects
configuration so /nodes/index_search continues targeting
/nodes/store_elasticsearch, while Elasticsearch-specific legacy paths target the
nested Elasticsearch page and OpenSearch-specific paths target the nested
OpenSearch page. Preserve the existing source paths and split the
provider-specific mappings accordingly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: c38bf400-e060-48af-bae1-7cb4d95155ae

📥 Commits

Reviewing files that changed from the base of the PR and between 75d3f81 and 132437a.

⛔ Files ignored due to path filters (12)
  • nodes/src/nodes/graph_arango/arango.svg is excluded by !**/*.svg
  • nodes/src/nodes/graph_hydradb/hydradb.svg is excluded by !**/*.svg
  • nodes/src/nodes/store_astra/astra_db.svg is excluded by !**/*.svg
  • nodes/src/nodes/store_atlas/mongodb.svg is excluded by !**/*.svg
  • nodes/src/nodes/store_chroma/chroma.svg is excluded by !**/*.svg
  • nodes/src/nodes/store_elasticsearch/elasticsearch.svg is excluded by !**/*.svg
  • nodes/src/nodes/store_elasticsearch/opensearch.svg is excluded by !**/*.svg
  • nodes/src/nodes/store_milvus/milvus.svg is excluded by !**/*.svg
  • nodes/src/nodes/store_pinecone/pinecone.svg is excluded by !**/*.svg
  • nodes/src/nodes/store_postgres/postgres.svg is excluded by !**/*.svg
  • nodes/src/nodes/store_qdrant/qdrant.svg is excluded by !**/*.svg
  • nodes/src/nodes/store_weaviate/weaviate.svg is excluded by !**/*.svg
📒 Files selected for processing (106)
  • nodes/src/nodes/graph_arango/IGlobal.py
  • nodes/src/nodes/graph_arango/IInstance.py
  • nodes/src/nodes/graph_arango/README.md
  • nodes/src/nodes/graph_arango/__init__.py
  • nodes/src/nodes/graph_arango/requirements.txt
  • nodes/src/nodes/graph_arango/services.json
  • nodes/src/nodes/graph_arango/utils.py
  • nodes/src/nodes/graph_hydradb/IGlobal.py
  • nodes/src/nodes/graph_hydradb/IInstance.py
  • nodes/src/nodes/graph_hydradb/README.md
  • nodes/src/nodes/graph_hydradb/__init__.py
  • nodes/src/nodes/graph_hydradb/hydradb_client.py
  • nodes/src/nodes/graph_hydradb/requirements.txt
  • nodes/src/nodes/graph_hydradb/services.json
  • nodes/src/nodes/store_astra/IEndpoint.py
  • nodes/src/nodes/store_astra/IGlobal.py
  • nodes/src/nodes/store_astra/IInstance.py
  • nodes/src/nodes/store_astra/README.md
  • nodes/src/nodes/store_astra/__init__.py
  • nodes/src/nodes/store_astra/astra_db.py
  • nodes/src/nodes/store_astra/requirements.txt
  • nodes/src/nodes/store_astra/services.json
  • nodes/src/nodes/store_atlas/IEndpoint.py
  • nodes/src/nodes/store_atlas/IGlobal.py
  • nodes/src/nodes/store_atlas/IInstance.py
  • nodes/src/nodes/store_atlas/README.md
  • nodes/src/nodes/store_atlas/__init__.py
  • nodes/src/nodes/store_atlas/atlas.py
  • nodes/src/nodes/store_atlas/requirements.txt
  • nodes/src/nodes/store_atlas/services.json
  • nodes/src/nodes/store_chroma/IEndpoint.py
  • nodes/src/nodes/store_chroma/IGlobal.py
  • nodes/src/nodes/store_chroma/IInstance.py
  • nodes/src/nodes/store_chroma/README.md
  • nodes/src/nodes/store_chroma/__init__.py
  • nodes/src/nodes/store_chroma/chroma.py
  • nodes/src/nodes/store_chroma/requirements.txt
  • nodes/src/nodes/store_chroma/services.json
  • nodes/src/nodes/store_elasticsearch/IEndpoint.py
  • nodes/src/nodes/store_elasticsearch/IGlobal.py
  • nodes/src/nodes/store_elasticsearch/IInstance.py
  • nodes/src/nodes/store_elasticsearch/README.md
  • nodes/src/nodes/store_elasticsearch/__init__.py
  • nodes/src/nodes/store_elasticsearch/constants.py
  • nodes/src/nodes/store_elasticsearch/elasticsearch/README.md
  • nodes/src/nodes/store_elasticsearch/elasticsearch_store.py
  • nodes/src/nodes/store_elasticsearch/opensearch/README.md
  • nodes/src/nodes/store_elasticsearch/opensearch_client.py
  • nodes/src/nodes/store_elasticsearch/requirements.txt
  • nodes/src/nodes/store_elasticsearch/services.elasticsearch.json
  • nodes/src/nodes/store_elasticsearch/services.opensearch.json
  • nodes/src/nodes/store_milvus/IEndpoint.py
  • nodes/src/nodes/store_milvus/IGlobal.py
  • nodes/src/nodes/store_milvus/IInstance.py
  • nodes/src/nodes/store_milvus/README.md
  • nodes/src/nodes/store_milvus/__init__.py
  • nodes/src/nodes/store_milvus/milvus.py
  • nodes/src/nodes/store_milvus/requirements.txt
  • nodes/src/nodes/store_milvus/services.json
  • nodes/src/nodes/store_pinecone/IEndpoint.py
  • nodes/src/nodes/store_pinecone/IGlobal.py
  • nodes/src/nodes/store_pinecone/IInstance.py
  • nodes/src/nodes/store_pinecone/README.md
  • nodes/src/nodes/store_pinecone/__init__.py
  • nodes/src/nodes/store_pinecone/pinecone.py
  • nodes/src/nodes/store_pinecone/requirements.txt
  • nodes/src/nodes/store_pinecone/services.json
  • nodes/src/nodes/store_postgres/IEndpoint.py
  • nodes/src/nodes/store_postgres/IGlobal.py
  • nodes/src/nodes/store_postgres/IInstance.py
  • nodes/src/nodes/store_postgres/README.md
  • nodes/src/nodes/store_postgres/__init__.py
  • nodes/src/nodes/store_postgres/postgres.py
  • nodes/src/nodes/store_postgres/requirements.txt
  • nodes/src/nodes/store_postgres/services.json
  • nodes/src/nodes/store_qdrant/IEndpoint.py
  • nodes/src/nodes/store_qdrant/IGlobal.py
  • nodes/src/nodes/store_qdrant/IInstance.py
  • nodes/src/nodes/store_qdrant/README.md
  • nodes/src/nodes/store_qdrant/__init__.py
  • nodes/src/nodes/store_qdrant/qdrant.py
  • nodes/src/nodes/store_qdrant/requirements.txt
  • nodes/src/nodes/store_qdrant/services.json
  • nodes/src/nodes/store_weaviate/IEndpoint.py
  • nodes/src/nodes/store_weaviate/IGlobal.py
  • nodes/src/nodes/store_weaviate/IInstance.py
  • nodes/src/nodes/store_weaviate/README.md
  • nodes/src/nodes/store_weaviate/__init__.py
  • nodes/src/nodes/store_weaviate/requirements.txt
  • nodes/src/nodes/store_weaviate/services.json
  • nodes/src/nodes/store_weaviate/weaviate.py
  • nodes/test/astra_db/test_convert_filter.py
  • nodes/test/atlas/test_isdeleted_filter.py
  • nodes/test/chroma/test_coerce_port.py
  • nodes/test/chroma/test_convert_filter_explicit_none.py
  • nodes/test/conftest.py
  • nodes/test/milvus/test_addchunks_return.py
  • nodes/test/milvus/test_filter_injection.py
  • nodes/test/milvus/test_search_semantic.py
  • nodes/test/pinecone/test_record_updates.py
  • nodes/test/qdrant/test_mark_deleted.py
  • nodes/test/qdrant/test_search_threshold.py
  • nodes/test/test_db_arango.py
  • nodes/test/test_db_hydradb.py
  • nodes/test/weaviate/test_record_updates.py
  • packages/docs/redirects.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review continued from previous batch...

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

🛑 Comments failed to post (51)
nodes/src/nodes/graph_arango/IGlobal.py (1)

184-204: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

affected_rows is silently zeroed out whenever a write query also returns rows.

_run_query_raw returns 'affected_rows': 0 if rows else affected — so an INSERT ... RETURN NEW (or any write with a RETURN clause) always reports affected_rows: 0, even though documents were actually written. This value flows unmodified into the answers lane JSON payload in IInstance.writeQuestions's EXECUTE handler, so an SDK caller relying specifically on affected_rows (rather than counting rows) gets a misleading zero for a successful write. len(rows) would be the more accurate affected count.

This requires two independent preconditions to surface (allow_execute=true, non-default, and a write query with RETURN), consistent with a previously flagged pattern in a sibling graph node, so Major rather than Critical severity applies here too.

🐛 Proposed fix
-        affected = _affected_rows(cursor)
-        return {'rows': rows, 'affected_rows': 0 if rows else affected}
+        affected = len(rows) if rows else _affected_rows(cursor)
+        return {'rows': rows, 'affected_rows': affected}

Based on learnings: "downgrade an issue's severity from Critical to Major if the bug can only be triggered when multiple independent preconditions align—such as a non-default config flag (e.g., allow_execute) being enabled, the executed query having a specific shape (e.g., write + RETURN), and downstream/caller logic relying on a specific returned field for retry logic (e.g., affected_rows)."

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def _run_query_raw(self, aql: str, *, max_runtime: float = QUERY_MAX_RUNTIME) -> Dict[str, Any]:
        """Execute a raw AQL statement without the ``_is_aql_safe`` gate.

        Used by the EXECUTE path where the caller has accepted the risk of running
        write AQL directly. Returns ``{'rows': [...], 'affected_rows': N}`` to mirror
        the SQL ``_executeRawQuery`` shape — ``affected_rows`` is derived from the
        cursor statistics when no rows are returned (e.g. INSERT without RETURN).

        Raises:
            arango.exceptions.ArangoError: Caught at the IInstance handler per precedent.
        """
        max_rows = self.max_execute_rows
        cursor = self.db.aql.execute(aql, count=False, max_runtime=max_runtime, memory_limit=self.QUERY_MEMORY_LIMIT)
        rows: List[Dict] = []
        for doc in cursor:
            # Check before appending (mirrors _run_query) so we never hold an extra row.
            if len(rows) >= max_rows:
                raise ValueError(f'EXECUTE query exceeded max_execute_rows={max_rows}')
            rows.append(doc)
        affected = len(rows) if rows else _affected_rows(cursor)
        return {'rows': rows, 'affected_rows': affected}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/graph_arango/IGlobal.py` around lines 184 - 204, Update
_run_query_raw so affected_rows reflects writes even when the query returns
rows: use len(rows) as the affected count when rows are present, while retaining
the cursor-derived affected count when no rows are returned. Keep the existing
row collection and max_execute_rows enforcement unchanged.

Source: Learnings

nodes/src/nodes/graph_arango/IInstance.py (1)

175-224: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

get_aql's input_schema doesn't document the limit parameter it actually accepts.

get_aql reads args.get('limit') (line 210) and get_data relies on this by passing limit through internally (line 104), but get_aql's own input_schema (lines 176-185) only declares question. An agent calling get_aql directly via the tool-invoke path has no documented way to know limit is supported.

♻️ Proposed fix
         input_schema={
             'type': 'object',
             'required': ['question'],
             'properties': {
                 'question': {
                     'type': 'string',
                     'description': 'Natural-language question to convert into an AQL query',
                 },
+                'limit': {
+                    'type': 'integer',
+                    'description': f'Row limit to embed in the generated AQL LIMIT clause (default 250, max {DEFAULT_MAX_EXECUTE_ROWS}).',
+                },
             },
         },
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    `@tool_function`(
        input_schema={
            'type': 'object',
            'required': ['question'],
            'properties': {
                'question': {
                    'type': 'string',
                    'description': 'Natural-language question to convert into an AQL query',
                },
                'limit': {
                    'type': 'integer',
                    'description': f'Row limit to embed in the generated AQL LIMIT clause (default 250, max {DEFAULT_MAX_EXECUTE_ROWS}).',
                },
            },
        },
        output_schema={
            'type': 'object',
            'properties': {
                'aql': {'type': 'string', 'description': 'The generated read-only AQL query.'},
                'valid': {'type': 'boolean', 'description': 'Whether a valid, safe AQL query was generated.'},
                'error': {'type': 'string', 'description': 'Error message if the generated AQL was unsafe.'},
                'answer': {
                    'type': 'string',
                    'description': 'LLM text response when the question is not a database query.',
                },
            },
        },
        description=(
            'Accepts a natural-language description and returns the equivalent read-only AQL query without executing it. Only use when the user explicitly asks to see the AQL — for actual data retrieval, use get_data instead.'
        ),
    )
    def get_aql(self, args):
        """Translate natural language to AQL without executing."""
        if not isinstance(args, dict):
            raise ValueError('Tool input must be a JSON object')
        question = args.get('question')
        if not question or not isinstance(question, str) or not question.strip():
            raise ValueError('"question" is required and must be a non-empty string')

        limit = _clamp_limit(args.get('limit'))
        result = self._buildAqlQuery(question.strip(), limit=limit)
        is_valid = _parse_is_valid(result.get('isValid', False))
        aql = result.get('query', '')

        # is_valid means the query passed the EXPLAIN-plan read-only gate in
        # _buildAqlQuery, so it is already known safe — no keyword re-check here.
        if is_valid and aql:
            return {'aql': aql, 'valid': True}
        # Not valid: an EXPLAIN-validation failure carries an 'error'; a genuine
        # off-topic question does not (its 'query' is the LLM's plain-text answer).
        if result.get('error'):
            return {'error': result['error'], 'aql': aql, 'valid': False}
        return {'answer': aql, 'valid': False}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/graph_arango/IInstance.py` around lines 175 - 224, Update the
get_aql tool’s input_schema to document the supported limit parameter alongside
question, including its expected numeric type and purpose, while preserving the
existing _clamp_limit(args.get('limit')) behavior.
nodes/src/nodes/graph_arango/README.md (1)

1-1: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Stale db_arango references left over from the rename.

The H1 title (line 1: "# db_arango") and the "View source" link (line 104, pointing to .../nodes/src/nodes/db_arango) still reference the old node name/path, even though this file now lives at nodes/src/nodes/graph_arango/README.md and the module path was updated elsewhere in this PR (services.json's path → nodes.graph_arango).

📝 Proposed fix
-# db_arango
+# graph_arango
-[<svg ...>View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/db_arango)
+[<svg ...>View source](https://github.com/rocketride-org/rocketride-server/tree/develop/nodes/src/nodes/graph_arango)

Also applies to: 104-104

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/graph_arango/README.md` at line 1, Update the README H1 and
its “View source” link to use the renamed graph_arango node and path, replacing
the stale db_arango references while preserving the existing link structure.
nodes/src/nodes/store_astra/astra_db.py (6)

62-90: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

api_endpoint is neither normalized nor validated as required.

The comment at line 70 claims stripping of spaces, http(s)://, and trailing slashes, but line 71 just does config.get('api_endpoint') with no processing at all — the comment is stale/misleading. Separately, unlike collection_name (regex-validated) and similarity (whitelist-validated), there's no check that api_endpoint is even present; a missing value silently becomes None and is passed straight into self.client.get_database(None, ...), producing an opaque astrapy error instead of a clear configuration error.

🐛 Proposed fix
-        # Remove leading and trailing spaces, leading http/https and :// and trailing slashes
-        self.api_endpoint = config.get('api_endpoint')
+        self.api_endpoint = config.get('api_endpoint')
+        if not self.api_endpoint:
+            raise Exception('Missing required config: api_endpoint')
+        self.api_endpoint = re.sub(r'^https?://', '', self.api_endpoint.strip()).rstrip('/')
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        config = Config.getNodeConfig(provider, connConfig)

        # Save our parameters
        self.collection_name = config.get('collection', '')

        if not re.match(r'^[A-Za-z0-9][A-Za-z0-9_]*$', self.collection_name):
            raise Exception(f'Invalid collection name: {self.collection_name}')

        self.api_endpoint = config.get('api_endpoint')
        if not self.api_endpoint:
            raise Exception('Missing required config: api_endpoint')
        self.api_endpoint = re.sub(r'^https?://', '', self.api_endpoint.strip()).rstrip('/')

        # check if the similarity matches astra configuration options
        similarity = config.get('similarity', 'cosine')
        if similarity in ['cosine', 'euclidean', 'dot_product']:
            self.similarity = similarity
        else:
            raise Exception(
                'The similarity metric you provided in the config.json does not match required astra configurations'
            )

        # Strip API key also
        self.application_token = config.get('application_token', None)
        if self.application_token is not None:
            self.application_token = self.application_token.strip()

        # Init the store
        self.client = DataAPIClient()
        self.database = self.client.get_database(self.api_endpoint, token=self.application_token)
        return
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_astra/astra_db.py` around lines 62 - 90, Update
api_endpoint initialization in the store setup to require a configured value,
normalize it by trimming whitespace, removing the http/https scheme and
surrounding trailing slashes as described by the existing comment, and validate
the resulting endpoint before passing it to get_database. Raise a clear
configuration error for missing or invalid endpoints, while preserving the
existing collection_name, similarity, and application_token handling.

147-200: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Unescaped user/path value injected into $regex filter.

docFilter.parent is inserted directly into {'$regex': docFilter.parent} with no escaping. Parent paths commonly contain regex metacharacters (., (, ), +, etc.) — e.g. . matches any character — so a path like /docs/report.pdf can incidentally match /docs/reportXpdf, silently returning wrong documents.

🐛 Proposed fix
         if docFilter.parent is not None:
-            filter_dict['meta.parent'] = {'$regex': docFilter.parent}
+            filter_dict['meta.parent'] = {'$regex': re.escape(docFilter.parent)}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_astra/astra_db.py` around lines 147 - 200, Escape the
user-supplied parent path before assigning it to the $regex filter in
_convertFilter. Use the standard regex-escaping utility so metacharacters in
docFilter.parent are matched literally, while preserving the existing parent
filter behavior and all other filter clauses.

276-299: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Missing embedding-null and non-zero-offset guards, unlike sibling stores.

Every other store's searchSemantic in this codebase (Elasticsearch, Milvus — see supplied graph context) explicitly checks if query.embedding is None: raise Exception(...) and rejects non-zero docFilter.offset with a clear error, since semantic/vector search paging by offset generally isn't supported. Astra's implementation does neither: a missing embedding will fail deep inside astrapy with an unclear error, and a non-zero offset is silently ignored (callers expecting pagination will always get the same first page).

🐛 Proposed fix
         filter_dict = self._convertFilter(docFilter)
+
+        # Check embedding
+        if query.embedding is None:
+            raise Exception('To use semantic search, you must bind to an embedding module')
+
+        # We cannot support non-zero offsets
+        if docFilter.offset:
+            raise BaseException('Non-zero offset is not supported in semantic searching')
Based on learnings, this mirrors the established pattern used elsewhere (e.g., `nodes/src/nodes/store_elasticsearch/elasticsearch_store.py` `searchSemantic`).
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def searchSemantic(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:
        """
        Semantic search.
        """
        if not self._doesCollectionExist():
            return []

        filter_dict = self._convertFilter(docFilter)

        # Check embedding
        if query.embedding is None:
            raise Exception('To use semantic search, you must bind to an embedding module')

        # We cannot support non-zero offsets
        if docFilter.offset:
            raise BaseException('Non-zero offset is not supported in semantic searching')

        # Combine both lexical and vector search
        cursor = self.database.get_collection(self.collection_name).find(
            filter_dict,
            sort={'$vector': query.embedding},  # Automatic embedding generation + search
            limit=docFilter.limit,
            projection={'$lexical': 0},
            include_similarity=True,
            include_sort_vector=True,
        )

        # Convert results to your Doc format
        docs = self._convertToDocs(cursor)

        return docs
🧰 Tools
🪛 ast-grep (0.44.1)

[warning] 285-292: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: self.database.get_collection(self.collection_name).find(
filter_dict,
sort={'$vector': query.embedding}, # Automatic embedding generation + search
limit=docFilter.limit,
projection={'$lexical': 0},
include_similarity=True,
include_sort_vector=True,
)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').

(xpath-injection-python)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_astra/astra_db.py` around lines 276 - 299, Update
searchSemantic to raise a clear exception when query.embedding is None and
reject any non-zero docFilter.offset before querying Astra. Preserve the
existing collection check and semantic search flow for valid embeddings with
offset zero.

318-357: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

getPaths bypasses the default soft-delete exclusion.

Unlike every other read path in this file, getPaths builds its own match_filter directly instead of going through _convertFilter, so it never adds meta.isDeleted: False. The README explicitly documents that soft-deleted chunks are "excluded from every query unless the filter explicitly requests them" — getPaths violates that, returning parent/objectId pairs for soft-deleted objects too.

🐛 Proposed fix
-        match_filter = {'meta.chunkId': 0}
+        match_filter = {'meta.chunkId': 0, 'meta.isDeleted': False}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def getPaths(self, parent: str | None = None, offset: int = 0, limit: int = 1000) -> Dict[str, str]:
        """
        More efficient implementation using aggregation to get distinct parent paths.
        """
        if not self._doesCollectionExist():
            return {}

        # Build match stage filter
        match_filter = {'meta.chunkId': 0, 'meta.isDeleted': False}

        if parent is not None:
            match_filter['meta.parent'] = parent

        # Use aggregation pipeline
        pipeline = [
            {'$match': match_filter},
            {
                '$group': {
                    '_id': '$meta.parent',
                    'objectId': {'$first': '$meta.objectId'},
                }
            },
            {'$skip': offset},
            {'$limit': limit},
        ]

        # Execute aggregation
        collection = self.database.get_collection(self.collection_name)
        results = collection.aggregate(pipeline)

        # Build paths dictionary
        paths: Dict[str, str] = {}
        for result in results:
            parent_path = result['_id']
            object_id = result['objectId']
            if parent_path and object_id:
                paths[parent_path] = object_id

        return paths
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_astra/astra_db.py` around lines 318 - 357, Update
getPaths to exclude soft-deleted records by adding the standard meta.isDeleted:
False condition to its aggregation match filter, while preserving the existing
parent filter, pagination, and path-building behavior.

358-435: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win

Critical: the near-zero-vector filter drops the boosted control/schema document for realistic embedding sizes.

is_valid_vector requires magnitude > 1e-6. The control-document workaround boosts a zero embedding to [1e-8] * len(embedding), giving magnitude = 1e-8 * sqrt(len(embedding)). For this to exceed 1e-6, len(embedding) must exceed 10,000 — but real embedding dimensions (384–3072) all fall far below that (e.g. at 1536 dims, magnitude ≈ 3.9e-7). So for essentially every real embedding model, valid_docs excludes the 'schema' control document, and it never gets inserted — silently defeating the special-case handling and, per doesCollectionExist's docstring, breaking the "control document" detection that base-class collection-fix logic depends on.

🐛 Proposed fix
-            def is_valid_vector(vector):
-                magnitude = sum(x * x for x in vector) ** 0.5
-                return magnitude > 1e-6
-
-            valid_docs = [doc for doc in documents if is_valid_vector(doc.get('$vector', [1]))]
+            def is_valid_vector(doc):
+                if doc.get('meta', {}).get('objectId') == 'schema':
+                    return True
+                vector = doc.get('$vector', [1])
+                magnitude = sum(x * x for x in vector) ** 0.5
+                return magnitude > 1e-6
+
+            valid_docs = [doc for doc in documents if is_valid_vector(doc)]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def addChunks(self, chunks: List[Doc], checkCollection: bool = True) -> None:
        """
        Add document chunks to the document store.
        """
        if not chunks:
            return

        if checkCollection and not self.createCollection(chunks):
            return

        collection = self.database.get_collection(self.collection_name)

        # Clear the documents list
        documents: List[Dict] = []

        # Clear the object id list for deletion
        objectIds: Dict = {}

        def flush():
            nonlocal documents
            nonlocal objectIds

            # First, delete existing documents if we have objectIds to delete
            if len(objectIds):
                # Delete all documents with these object IDs
                for objectId in objectIds.keys():
                    collection.delete_many({'meta.objectId': objectId})

            def is_valid_vector(doc):
                if doc.get('meta', {}).get('objectId') == 'schema':
                    return True
                vector = doc.get('$vector', [1])
                magnitude = sum(x * x for x in vector) ** 0.5
                return magnitude > 1e-6

            valid_docs = [doc for doc in documents if is_valid_vector(doc)]

            # Insert the new documents
            if len(valid_docs):
                collection.insert_many(valid_docs)

            # Clear them
            objectIds = {}
            documents = []

        # For each document, check if we need to delete existing chunks
        for chunk in chunks:
            # If we are writing chunk 0, then delete the object
            if not chunk.metadata.chunkId:
                # Save this object id to be deleted
                objectIds[chunk.metadata.objectId] = True

        # For each document, prepare the data for insertion
        for chunk in chunks:
            # Get the embedding
            embedding = chunk.embedding

            # If we do not have an embedding
            if embedding is None:
                raise Exception('No embedding in document')

            # Handle control document with zero vectors
            if chunk.metadata.objectId == 'schema' and all(x == 0 for x in embedding):
                embedding = [1e-8] * len(embedding)

            # Create the document structure for AstraDB
            document = {
                '_id': str(uuid4()),
                '$vector': DataAPIVector(embedding),
                'content': chunk.page_content,
                'meta': chunk.metadata.__dict__,  # Convert metadata to dict
            }

            documents.append(document)

            if len(documents) >= 500:
                flush()

        # Flush any remaining documents
        flush()
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_astra/astra_db.py` around lines 358 - 435, Update the
near-zero-vector validation inside addChunks.flush so the schema control
document created by the zero-embedding workaround is retained for insertion.
Ensure the validity check or schema-specific handling accepts that boosted
control vector while continuing to filter genuinely invalid vectors for other
documents, preserving doesCollectionExist control-document detection.

490-529: 🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift

render loads the entire object into memory in one shot, contradicting its own comment and diverging from sibling stores.

The comment above render (lines 490-493) says chunks are gathered "one group at a time (renderChunkSize)" and the callback is invoked per group — matching how Atlas, Postgres, and Milvus actually implement render (looping with skip/limit by renderChunkSize, calling callback per batch). Astra's implementation instead fetches all chunks for the object unconditionally, sorts them all in memory, joins them into one string, and calls callback exactly once. There is no renderChunkSize attribute on this Store at all. For objects with tens of thousands of chunks (explicitly called out as the target case), this risks large memory spikes/OOM and is inconsistent with the rest of the store fleet's render contract.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_astra/astra_db.py` around lines 490 - 529, Update
Store.render to process object chunks in batches using the store’s
renderChunkSize configuration, rather than collecting, sorting, joining, and
callbacking the entire object at once. Preserve chunk ordering by fetching each
batch in chunkId order, invoke callback once per batch, and add or reuse the
appropriate renderChunkSize attribute consistent with sibling stores.
nodes/src/nodes/store_astra/README.md (1)

84-99: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Generated "Source" link still points to the pre-rename astra_db path.

The generated block links to nodes/src/nodes/astra_db, but this node now lives at nodes/src/nodes/store_astra. As per coding guidelines, content in this block must not be hand-edited since it is "regenerated by nodes:docs-generate" — please re-run the docs generator against the renamed path before merge so this link resolves correctly.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_astra/README.md` around lines 84 - 99, Regenerate the
documentation for the renamed store_astra node using the nodes:docs-generate
workflow so the generated Source link points to nodes/src/nodes/store_astra
instead of the obsolete astra_db path. Do not manually edit the generated block.

Source: Coding guidelines

nodes/src/nodes/store_atlas/atlas.py (5)

236-249: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Redundant, inconsistent collection-existence re-check in searchSemantic.

Line 238 already returns [] if the collection is missing. Line 242 re-checks with self.doesCollectionExist(query.embedding_model) and raises instead — but _doesCollectionExist() (L131-135) takes no parameters and never inspects an embedding model, so the comment "Validate embedding model matches collection" doesn't reflect what's actually being checked. This looks like leftover/dead logic that produces confusing, inconsistent error behavior (silent [] vs raised exception) for what is effectively the same failure condition.

🐛 Suggested cleanup
-        if not self.doesCollectionExist():
-            return []
-
-        # Validate embedding model matches collection
-        if not self.doesCollectionExist(query.embedding_model):
-            raise Exception('Collection does not exist or model mismatch')
-
-        if query.embedding is None:
+        if not self.doesCollectionExist():
+            return []
+
+        if query.embedding is None:
             raise Exception('Embedding required for semantic search')
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    def searchSemantic(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:
        """Perform semantic search using Atlas Vector Search."""
        if not self.doesCollectionExist():
            return []

        if query.embedding is None:
            raise Exception('Embedding required for semantic search')

        if docFilter.offset:
            raise Exception('Non-zero offset not supported in semantic search')
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_atlas/atlas.py` around lines 236 - 249, Remove the
redundant model-parameter collection check and its misleading validation comment
from searchSemantic. Keep the initial doesCollectionExist() guard returning []
for missing collections, while preserving the separate embedding and offset
validations.

270-276: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

UnboundLocalError when the aggregation raises OperationFailure.

results is only assigned inside the try block. If self.collection.aggregate(pipeline) raises OperationFailure, the except branch only logs the error and falls through — results is never bound, and the subsequent return self._convertToDocs(results, ...) crashes with UnboundLocalError instead of failing gracefully.

🐛 Proposed fix
+        results = []
         try:
             results = list(self.collection.aggregate(pipeline))
             logger.info(f'Results: {results}')
         except OperationFailure as e:
             logger.error(f'OperationFailure: {e}')
+            return []
 
         return self._convertToDocs(results, include_score=True)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        results = []
        try:
            results = list(self.collection.aggregate(pipeline))
            logger.info(f'Results: {results}')
        except OperationFailure as e:
            logger.error(f'OperationFailure: {e}')
            return []

        return self._convertToDocs(results, include_score=True)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_atlas/atlas.py` around lines 270 - 276, Initialize
results to a safe empty collection before the try block, or return an
appropriate empty-document result directly from the OperationFailure handler, so
_convertToDocs receives a defined value when aggregate fails. Preserve the
existing successful aggregation and logging behavior.

294-316: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file outline =="
ast-grep outline nodes/src/nodes/store_atlas/atlas.py --view expanded || true

echo "== relevant sections =="
cat -n nodes/src/nodes/store_atlas/atlas.py | sed -n '260,330p'
printf '\n---\n'
cat -n nodes/src/nodes/store_atlas/atlas.py | sed -n '420,495p'

echo "== searches for parent/docFilter source =="
rg -n "def getPaths|parent|docFilter|_convertFilter|get_paths|docsFilter|storeCollection|getCollection|filter_conditions" nodes/src/nodes/store_atlas/atlas.py

echo "== imports =="
cat -n nodes/src/nodes/store_atlas/atlas.py | sed -n '1,80p'

Repository: rocketride-org/rocketride-server

Length of output: 15588


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate schema/store definitions =="
git ls-files | rg 'ai/common/schema\.py$|ai/common/store\.py$|schema\.py$|store\.py$' || true

echo "== DocFilter occurrences =="
rg -n "class DocFilter|DocFilter|class DocumentStoreBase|def getPaths|getDocs|searchKeyword|searchSemantic|get\(" .

echo "== schema section around DocFilter =="
for f in $(git ls-files | rg 'ai/common/schema\.py$|ai/common/store\.py$'); do
  echo "--- $f ---"
  cat -n "$f" | sed -n '1,220p'
done

Repository: rocketride-org/rocketride-server

Length of output: 50389


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== candidate schema/store files =="
git ls-files | rg '(^|/)schema\.py$|(^|/)store\.py$' | sed -n '1,120p'

echo "== focused DocFilter definitions =="
rg -n "class DocFilter|DocFilter =|parent:.*str|parent.*str|metadata" ai/common/schema.py ai/common/store.py 2>/dev/null | sed -n '1,160p' || true

echo "== focused schema/store snippets =="
for f in ai/common/schema.py.ai/common/store.py; do
  if [ -f "$f" ]; then
    echo "--- $f ---"
    cat -n "$f" | sed -n '1,260p'
  fi
done

Repository: rocketride-org/rocketride-server

Length of output: 462


🏁 Script executed:

#!/bin/bash
set -euo pipefail

for f in packages/ai/src/ai/common/store.py packages/ai/src/ai/common/schema.py; do
  if [ -f "$f" ]; then
    echo "--- $f ---"
    wc -l "$f"
    cat -n "$f" | sed -n '1,260p'
  else
    echo "MISSING: $f"
  fi
done

echo "== imports and API paths mentioning getPaths / DocFilter =="
rg -n "from ai\.common\.schema import|import .*schema|DocFilter|getPaths\(|searchKeyword|searchSemantic|\.get\(" packages nodes | sed -n '1,240p'

Repository: rocketride-org/rocketride-server

Length of output: 38951


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== shell-api DocFilter/type definitions =="
if [ -f packages/shell-api/versions/v0.d.ts ]; then
  cat -n packages/shell-api/versions/v0.d.ts | sed -n '70,100p;150,185p'
fi

echo "== TypeScript/TSX parent-field usages =="
rg -n "DocFilter|filter.*parent|parent.*filter|getPaths|\bparent\b" packages shared-ui 2>/dev/null | sed -n '1,260p' || true

echo "== Python search/store wrappers around getPaths/search calls =="
rg -n "getPaths\(|searchKeyword\(|searchSemantic\(|DocFilter\(" nodes/src packages/ai/src packages/client-python/src 2>/dev/null | sed -n '1,240p' || true

Repository: rocketride-org/rocketride-server

Length of output: 40644


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== ai/common/schema.py files =="
git ls-files | rg 'packages/.*/src/ai/common/schema\.py$|ai/common/schema\.py$' || true

echo "== ai/common/schema.py doc filter excerpt =="
for f in $(git ls-files | rg 'packages/.*/src/ai/common/schema\.py$|ai/common/schema\.py$'); do
  echo "--- $f ---"
  rg -n "class DocFilter|parent|Field\(|pydantic|BaseModel" "$f" | sed -n '1,180p'
  if rg -n "class DocFilter" "$f" >/dev/null; then
    line=$(rg -n "^class DocFilter" "$f" | head -1 | cut -d: -f1 || true)
    if [ -n "${line:-}" ]; then
      start=$(( line >= 1 ? line-20 : 1 ))
      end=$(( line + 180 ))
      cat -n "$f" | sed -n "${start},${end}p"
    fi
  fi
done

echo "== schema converter around parent doc filter =="
for f in $(git ls-files | rg 'packages/.*/src/ai/common/schema\.py$|ai/common/schema\.py$'); do
  rg -n "docFilter|DocFilter|parent|convert|toDict|fromDict|search" "$f" | sed -n '1,240p'
  echo "--- $f converter sections ---"
  rg -n "def .*doc.*filter|def .*filter|doc_filters|filter_schema|Question|parent" "$f" | while read -r match; do
    line=$(echo "$match" | cut -d: -f1)
    echo "at line $line"
    start=$(( line >= 40 ? line-40 : 1 ))
    end=$(( line + 120 ))
    cat -n "$f" | sed -n "${start},${end}p"
    echo "------------"
  done || true
done

echo "== shell api doc filter implementation if present =="
fd -a 'DocFilter\.(ts|js)$|DocFilter\.d\.ts$|schema' packages/shell-api packages/shared-ui | sed -n '1,80p' || true

Repository: rocketride-org/rocketride-server

Length of output: 348


Escape parent values before MongoDB regex filtering.

getPaths() and _convertFilter() pass parent/docFilter.parent directly into $regex filters. If those values can originate from external/tool input, escaping them (e.g. re.escape(...)) prevents regex-injection-style filter bypass and costly pattern backtracking.

🧰 Tools
🪛 ast-grep (0.44.1)

[warning] 302-302: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: self.collection.find(filter_conditions)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').

(xpath-injection-python)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_atlas/atlas.py` around lines 294 - 316, Escape
externally supplied parent filter values before constructing MongoDB `$regex`
conditions in both `getPaths()` and `_convertFilter()`. Apply `re.escape(...)`
to `parent` and `docFilter.parent` while preserving the existing
case-insensitive matching behavior and filter structure.

318-353: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Existing documents are deleted before every chunk's embedding is validated.

objectIds are computed and deleted (L327-334) before the loop that raises Exception('No embedding in document') (L341-342) for any chunk missing an embedding. If a later chunk in the batch lacks an embedding, the pre-existing documents for that objectId have already been removed and the exception aborts before the replacement documents are inserted — leaving the object with no data in the store.

Validate that every chunk has an embedding before deleting the pre-existing documents for the same object IDs.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_atlas/atlas.py` around lines 318 - 353, Update
addChunks to validate that every chunk has a non-None embedding before computing
and deleting existing object IDs. Move the embedding validation ahead of the
objectIds deletion block, while preserving the existing exception behavior and
document preparation flow.

351-352: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win

sys.getsizeof(doc) does not measure real document size.

sys.getsizeof only returns the shallow size of the dict container, not the size of the nested embedding list or content string it holds. The payload_limit-based flush branch (L356) is effectively driven only by the 500-document count check, since sum_size will almost never reflect actual payload bytes — risking oversized batches when embeddings/content are large.

Consider estimating size via len(content.encode()) + len(embedding) * 8 (or BSON-encoding the doc) instead of sys.getsizeof.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_atlas/atlas.py` around lines 351 - 352, Replace the
shallow sys.getsizeof(doc) calculation in the document batching loop with an
estimate of the serialized payload that includes content bytes and embedding
element storage, or BSON-encode the document for an exact measure. Keep
accumulating the estimate in sum_size so the existing payload_limit flush
condition reflects actual document size.
nodes/src/nodes/store_atlas/README.md (1)

74-77: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Generated "View source" link still points to the old atlas path.

The link targets .../tree/develop/nodes/src/nodes/atlas, but the node now lives under nodes/src/nodes/store_atlas. Since this is inside the ROCKETRIDE:GENERATED:PARAMS block, re-run nodes:docs-generate against the renamed directory rather than hand-editing it.

As per coding guidelines, "do not hand-edit content between <!-- ROCKETRIDE:GENERATED:PARAMS START --> and <!-- ROCKETRIDE:GENERATED:PARAMS END --> as it is regenerated by nodes:docs-generate" — please regenerate rather than patch this line directly.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_atlas/README.md` around lines 74 - 77, Regenerate the
documentation for the renamed store_atlas node using nodes:docs-generate so the
generated “View source” link targets nodes/src/nodes/store_atlas. Do not
manually edit content within the ROCKETRIDE:GENERATED:PARAMS block.

Source: Coding guidelines

nodes/src/nodes/store_chroma/chroma.py (4)

245-253: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

chromadb collection query cosine distance range hnsw:space

💡 Result:

In ChromaDB, when you configure a collection with hnsw:space: "cosine", the system calculates and returns "cosine distance," not standard cosine similarity [1][2]. Cosine distance is derived from cosine similarity using the formula: Distance = 1.0 - Cosine Similarity [3][4][2]. Because of this definition, the range of values returned by ChromaDB for cosine distance is: - 0.0: Represents identical vectors (maximum similarity) [4][2][5]. - 1.0: Represents orthogonal vectors (no similarity) [4][5]. - 2.0: Represents opposite vectors (maximum dissimilarity) [2]. Key implementation details include: - Configuration: You specify the distance metric at collection creation time using configuration={"hnsw": {"space": "cosine"}} or the legacy metadata approach metadata={"hnsw:space": "cosine"} [1][6][2]. This setting cannot be changed after the collection is created [6]. - Interaction: When querying a collection configured with cosine, the distances field in the query result will contain these cosine distance values [1][7][8]. - Best Practices: Since ChromaDB returns distance (where lower is closer/better), you should interpret the results accordingly [9][2]. If your application requires a traditional similarity score (ranging from -1 to 1), you can convert the ChromaDB distance back to similarity by calculating (1.0 - distance) [2][5].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="nodes/src/nodes/store_chroma/chroma.py"
if [ -f "$file" ]; then
  echo "== file size =="
  wc -l "$file"
  echo
  echo "== relevant excerpts =="
  sed -n '1,120p' "$file" | cat -n
  echo
  sed -n '120,290p' "$file" | cat -n -v
  echo
  echo "== cosine/distances occurrences =="
  rg -n "similarity|distances|cosine|embedding|query" "$file" -C 3
else
  echo "missing $file"
  git ls-files | rg 'chroma\.py$'
fi

Repository: rocketride-org/rocketride-server

Length of output: 20134


Normalize Chroma cosine scores in the same direction as the distance.

Chroma returns distances for cosine as 1 - cosine_similarity in [0, 2], so better matches have lower distances. The current mapping (distance + 1) / 2 makes a perfect match score 0.5 and an opposite vector score 1.5; apply the score threshold self.threshold_search with this score ordering (or convert to 1 - distance/2 before thresholding).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_chroma/chroma.py` around lines 245 - 253, Update the
cosine branch in the score calculation around self.similarity to convert
Chroma’s distance into a similarity score using 1 - distance / 2, so lower
distances produce higher scores. Apply self.threshold_search to this normalized
score instead of the hardcoded 0.20 threshold, while preserving the existing
non-cosine scoring path.

374-378: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate file =="
git ls-files | grep -F 'nodes/src/nodes/store_chroma/chroma.py' || true

echo "== outline relevant file =="
ast-grep outline nodes/src/nodes/store_chroma/chroma.py --match getPaths --view expanded || true
ast-grep outline nodes/src/nodes/store_chroma/chroma.py --match _convertFilter --view expanded || true

echo "== relevant sections =="
sed -n '330,410p' nodes/src/nodes/store_chroma/chroma.py
echo "---"
sed -n '1,180p' nodes/src/nodes/store_chroma/chroma.py

echo "== search related filter usage =="
rg -n "chunkId|filter_dict|getPaths|_convertFilter|parent" nodes/src/nodes/store_chroma/chroma.py

Repository: rocketride-org/rocketride-server

Length of output: 11643


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== chromadb dependency versions =="
fd -a 'requirements.txt|pyproject.toml|poetry.lock|Pipfile.lock' . | sed 's#^\./##' | while read -r f; do
  if grep -qiE 'chromad( |b|b\.)|chroma' "$f"; then
    echo "--- $f"
    grep -n -iE 'chromad( |b|b\.)|chroma' "$f" || true
  fi
done

echo "== relevant filter helper =="
sed -n '180,230p' nodes/src/nodes/store_chroma/chroma.py

echo "== relevant add chunk storage =="
sed -n '420,590p' nodes/src/nodes/store_chroma/chroma.py

echo "== tests/usages of getPaths and chunkId =="
rg -n "getPaths|chunkId|chunk\":|'chunk'|\"chunk\"" . | head -200

Repository: rocketride-org/rocketride-server

Length of output: 35392


🌐 Web query:

ChromaDB 0.5 Python client get where metadata multiple keys top-level where two keys $and

💡 Result:

To filter by multiple metadata keys using the $and operator in the ChromaDB Python client, you must provide a dictionary to the where argument with the $and key associated with a list of condition objects [1][2]. Each condition object in the list should define a single metadata field filter [2]. Example structure for filtering by two keys: collection.get( where={ "$and": [ {"key1": "value1"}, {"key2": {"$gte": 10}} ] }) Key points for implementation: 1. Use a List for $and: The $and operator requires a list of dictionaries [1][2]. Each dictionary in that list must contain exactly one field condition [2]. 2. Comparison Operators: You can mix simple equality checks (e.g., {"key1": "value1"}) with more complex operators like $gte, $lte, $contains, etc. (e.g., {"key2": {"$gte": 10}}) [1][3][4]. 3. Alternative (K-Expressions): ChromaDB also supports a cleaner syntax using the K expression builder, which handles the $and construction automatically and often reduces syntax errors [2][5]. Example using K-Expressions: from chromadb import K # This compiles to the same $and JSON structure as above results = collection.get( where=(K("key1") == "value1") & (K("key2") >= 10)) Important Note: Do not attempt to put multiple keys directly inside a single dictionary for the where argument (e.g., where={"key1": "value1", "key2": "value2"}), as this is not the standard way to guarantee logical AND behavior across multiple conditions in older versions and may cause unexpected results or errors [6]. Always use the explicit $and list format or the K-expression builder [1][2][5].

Citations:


Use chunkId for the getPaths filter and wrap multi-condition filters in $and.

getPaths filters against chunk, but these chunks are stored/referenced as chunkId, so the query matches the wrong metadata key. Use {'chunkId': {'$eq': 0}} and wrap it with $and when adding the parent condition.

🧰 Tools
🪛 ast-grep (0.44.1)

[info] 377-377: use jsonify instead of json.dumps for JSON output
Context: json.dumps(parent, ensure_ascii=False)
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_chroma/chroma.py` around lines 374 - 378, Update the
getPaths filter construction to use the chunkId metadata key with an equality
condition for 0. When parent is provided, combine the chunkId and parent
conditions under a $and wrapper, preserving the existing parent serialization.

498-503: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate and inspect the relevant file and methods without executing repo code.
printf 'Files matching chroma.py:\n'
fd -a 'chroma\.py$' . || true

printf '\nRelevant source excerpts:\n'
if [ -f nodes/src/nodes/store_chroma/chroma.py ]; then
  wc -l nodes/src/nodes/store_chroma/chroma.py
  sed -n '460,535p' nodes/src/nodes/store_chroma/chroma.py | cat -n
fi

printf '\nChroma dependency/version references:\n'
for f in pyproject.toml requirements.txt requirements-dev.txt setup.py setup.cfg Pipfile poetry.lock uv.lock; do
  [ -f "$f" ] && { echo "--- $f"; rg -n "chromadb|chroma" "$f" || true; }
done

printf '\nSearch Chroma update/get usage in repo:\n'
rg -n "collectionObj\.(get|update)|\.get\(.*include|include=\['metadatas'\]|update\(ids=|new_metadata" nodes/src/nodes/store_chroma/chroma.py

Repository: rocketride-org/rocketride-server

Length of output: 4343


🌐 Web query:

ChromaDB collection get include metadatas result dict shape ids metadatas

💡 Result:

In ChromaDB, the collection.get method returns data in a column-oriented format (a dictionary of lists) [1][2]. The results are returned as a GetResult object [3][4]. When you specify include=["metadatas"] (or include=["documents", "metadatas"], which is the default), the structure of the returned dictionary is as follows [3][5]: - ids: A flat list of strings, representing the IDs of the retrieved records [1][6]. - metadatas: An optional list of dictionaries, where each dictionary corresponds to the metadata of the record at the same index in the ids list [3][1][2]. - documents: An optional list of strings (if included), corresponding to the record at the same index [3][1][2]. Because the data is column-oriented, all returned lists (ids, metadatas, documents, etc.) are aligned by index [1][2]. For example, ids[i], metadatas[i], and documents[i] all refer to the same record [1][2]. Example usage: result = collection.get(include=["documents", "metadatas"]) # Accessing data by index for id, doc, meta in zip(result["ids"], result["documents"], result["metadatas"]): print(id, doc, meta) If a field is not requested in the include parameter, it will not be present in the returned dictionary [3][1]. Note that ids are always returned, regardless of the include parameter [1][2].

Citations:


🌐 Web query:

ChromaDB collection update API ids metadatas where new_metadata

💡 Result:

In ChromaDB, you update the metadata of existing records using the update method on a collection [1][2]. When calling update, passing the ids parameter along with the new metadata dictionary in the metadatas parameter will overwrite the existing metadata for those specific records [3][4]. Key points regarding the update process: 1. Metadata Replacement: The update operation replaces the entire metadata dictionary for the specified IDs [3][4]. If you wish to retain existing keys and only add or modify specific ones, you must first retrieve the current metadata using get, update the dictionary in your application code, and then pass the complete, modified dictionary back to the update method [5]. 2. Selective Updating: You do not need to provide documents or embeddings when updating metadata [2]. If you only provide ids and metadatas, ChromaDB will update the metadata in place without re-generating embeddings or altering the document content [2]. 3. Upsert Alternative: You can also use the upsert method, which behaves identically to update for existing IDs but will create a new record if the provided ID does not already exist [1][2]. Example usage: # To update metadata without overwriting existing fields: # 1. Get the current metadata record = collection.get(ids=["doc1"], include=["metadatas"]) new_metadata = record["metadatas"] new_metadata["new_key"] = "new_value" # Add or update keys # 2. Update the record collection.update( ids=["doc1"], metadatas=[new_metadata]) Note that if you are looking to update the metadata of the collection itself (as an object) rather than individual records, you must use the collection.modify(metadata={...}) method, which also fully replaces the collection's metadata [6][7][5].

Citations:


Fix soft-delete/reactivate metadata updates. get(include=['metadatas']) returns columns (ids, metadatas), so both methods currently iterate keys like ids/metadatas and pass unsupported where=/new_metadata= kwargs to update(). Build parallel ids and metadatas lists, modify metadata['isDeleted'], then call self.collectionObj.update(ids=ids, metadatas=metadatas) while preserving existing metadata keys. Mirror this change in markActive.

📍 Affects 1 file
  • nodes/src/nodes/store_chroma/chroma.py#L498-L503 (this comment)
  • nodes/src/nodes/store_chroma/chroma.py#L522-L527
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_chroma/chroma.py` around lines 498 - 503, The
soft-delete and reactivation flows in markDeleted and markActive incorrectly
iterate Chroma result keys and call update with unsupported arguments. At both
affected sites in nodes/src/nodes/store_chroma/chroma.py (lines 498-503 and
522-527), build parallel ids and metadatas lists from the returned result
columns, update each metadata object’s isDeleted value while preserving all
existing keys, then call self.collectionObj.update with ids and metadatas.

567-591: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file locations =="
fd -a 'chroma.py$' . | sed 's#^\./##'

echo "== outline =="
ast-grep outline nodes/src/nodes/store_chroma/chroma.py --match render --view expanded || true

echo "== relevant source =="
sed -n '530,610p' nodes/src/nodes/store_chroma/chroma.py | cat -n

echo "== chroma get/include usages =="
rg -n "collectionObj\.get|get\(.*include=" nodes/src/nodes/store_chroma -S -C 3 || true

echo "== imports and setup =="
sed -n '1,80p' nodes/src/nodes/store_chroma/chroma.py | cat -n
sed -n '480,570p' nodes/src/nodes/store_chroma/chroma.py | cat -n

echo "== python/chroma docs from installed package if available =="
python3 - <<'PY'
try:
    import chromadb
    print("chromadb", getattr(chromadb, "__version__", None))
    from chromadb.config import Settings
    import inspect
    print("DocumentAPI.get signature:")
    print(inspect.signature(chromadb.api.models.Collection.Collection.get))
    help(chromadb.api.models.Collection.Collection.get)
except Exception as e:
    print("chromadb probe failed:", type(e).__name__, str(e))
PY

Repository: rocketride-org/rocketride-server

Length of output: 15507


🌐 Web query:

ChromaDB Python Collection.get include metadatas documents return format results ids metadatas documents

💡 Result:

In ChromaDB, the collection.get method returns results in a column-oriented format [1][2][3]. The results are returned as a GetResult dictionary, which organizes each requested field (such as documents, metadatas, or embeddings) into its own list [1][3][4]. The structure of the returned object is as follows: - ids: Always returned as a list of strings [2][3]. - documents: Returned as a list of strings (or nulls) if requested [2][3][4]. - metadatas: Returned as a list of dictionaries if requested [2][3][4]. - embeddings: Returned as a list of lists if requested [2][3][4]. Crucially, because the format is column-oriented, the data is aligned by index [2][3]. This means that for any index 'i', results['ids'][i], results['documents'][i], and results['metadatas'][i] all refer to the same record [2][3]. You can control which fields are included in this output using the include parameter [1][3]. By default, if the include parameter is not specified, ChromaDB typically returns both "documents" and "metadatas" [1][3]. Because these lists are aligned, it is common practice to process the results by zipping them together [1][2][3]: result = collection.get(include=["documents", "metadatas"]) # Zip aligned columns to process records for id, document, metadata in zip(result["ids"], result["documents"], result["metadatas"]): print(id, document, metadata)

Citations:


🌐 Web query:

site:docs.trychroma.com reference/python/collection get include metadatas documents None

💡 Result:

When you perform a.get operation in ChromaDB and receive None for fields like metadatas or documents, it is not an error [1]. This behavior occurs because ChromaDB does not return all available data by default to optimize performance and reduce memory usage [1][2]. To retrieve specific fields, you must explicitly include them using the include parameter [3][2]. How to resolve this: You can specify which fields to return by passing a list of strings to the include parameter in your.get call [3][2]. For example, if you want to include both documents and metadatas, you should structure your code as follows: results = collection.get( include=["documents", "metadatas"]) Key points: 1. Default Behavior: By default, the.get method returns documents and metadatas [3][2]. If you explicitly set include to something else, or if the data was not stored with those fields, they may appear as None in the result [1][2]. 2. Available Fields: Valid strings for the include parameter include "documents", "metadatas", "embeddings", "uris", and "distances" (though "distances" is typically used with.query) [3][2]. 3. IDs are Always Returned: Note that ids are always returned regardless of the include parameter [2]. 4. Troubleshooting: If you find fields are None even when you expect data, ensure that the data was actually stored for those records in the collection and that you are including them in the include list [1][2].

Citations:


🌐 Web query:

site:github.com/chroma-core/chroma GetResult metadatas None includes_metadata

💡 Result:

The GetResult object is a TypedDict used by the ChromaDB vector database to structure the output of collection.get and query operations [1][2][3]. The metadatas field within this object is an Optional[List[Metadata]] [1][3]. When the metadatas field appears as None in a GetResult object, it typically signifies one of the following scenarios: 1. Data Not Requested: The include parameter in the get or query operation did not specify "metadatas" as a field to be included in the response [2][4]. In such cases, the field is often returned as None or omitted, depending on the implementation [5][6]. 2. Records Lack Metadata: If "metadatas" was explicitly included in the include parameter, but the returned records do not have associated metadata, the field may return None for those specific entries or the list itself may be None/empty [1][4]. 3. API Transport: When using the ChromaDB FastAPI client, the server may explicitly pass back None for fields that were not included in the request or are absent, which is then mapped to the GetResult object [4]. To verify which fields were included in a specific operation, the GetResult object also contains an included field (e.g., included: Include), which lists the fields that were requested and returned in the payload [1][7][8]. This field was introduced to clarify the status of non-included fields, preventing ambiguity when a field like embeddings or metadatas is returned as None [7][6][8].

Citations:


🌐 Web query:

ChromaDB Python get result metadata is None list when include metadatas

💡 Result:

If you are seeing None values in your metadatas list when using ChromaDB's get or query methods, it is typically due to one of the following reasons: 1. Records lack metadata: If a specific document in your collection does not have associated metadata, ChromaDB will return None for that entry [1]. In a returned results dictionary, the metadatas field is a list where each index corresponds to the document at the same index in the ids or documents lists [2][3]. If no metadata exists for a record, the corresponding entry in the metadatas list will be None [1]. 2. Include parameter configuration: Ensure that "metadatas" is explicitly included in the include argument of your get or query call [2][4]. While "metadatas" is often included by default, it is best practice to specify it if you are encountering unexpected None results to ensure the data is fetched from the database [2][5]. Example of correct usage: result = collection.get(include=["documents", "metadatas"]) 3. Data insertion issues: Verify that you correctly assigned metadata when adding documents to the collection [6]. If the metadata was not correctly passed (e.g., using an incorrect list/dictionary structure during bulk insertion), it may result in None values later [6]. Note that ChromaDB returns data in a column-oriented format [2][3]. If you have multiple results, ensure you are iterating over the aligned lists (e.g., using zip) to correctly associate IDs, documents, and their respective metadata [2][3]. If you still see None unexpectedly, inspect the raw data added to the collection to confirm that the records were stored with the expected metadata dictionary [6].

Citations:


🌐 Web query:

ChromaDB Python httpClient GetResult metadatas None when include metadatas

💡 Result:

If you are receiving None for metadatas in ChromaDB when explicitly requesting them via the include parameter in the Python HttpClient, it is likely due to one of the following reasons: 1. Absence of Metadata in Stored Records: If the specific records retrieved do not have any metadata associated with them, ChromaDB will return None (or null) for that specific record in the results list [1]. It is important to note that metadata is optional; if a document was added without metadata, the corresponding entry in the returned list will be None [1]. 2. Incorrect Include Parameter Format: Ensure the include parameter is passed as a list of strings [2][3]. For the get method, the include list should contain "metadatas" [2][4]. Example of correct usage: collection.get( ids=["your_id"], include=["documents", "metadatas"]) 3. Client-Server Deserialization or Version Issues: There have been historical reports of inconsistencies between different client versions (e.g., Python vs. TypeScript/JavaScript) or during API serialization where metadata appears as null [5][6]. If you are using the HttpClient, ensure your client version is up-to-date and matches the version of the Chroma server [5]. 4. Checking the Data Source: Verify that metadata was indeed stored correctly during the initial add or upsert operation [7][5]. You can debug this by checking a single, known record to confirm that metadata exists in the database [1]. If the returned metadatas list contains None values, it confirms that the request was successful but that the specific documents retrieved simply have no metadata stored in the collection [1]. If the entire metadatas field is missing (None instead of a list), double-check that "metadatas" is correctly included in the list passed to the include argument [2][7][4].

Citations:


🌐 Web query:

"def get" "include" "_include_metadatas" "metadatas" chromadb api collection get

💡 Result:

In the ChromaDB Python API, the get method on a collection uses the include parameter to specify which fields should be returned for the retrieved records [1][2]. The get method is defined as follows: python collection.get( ids=None, where=None, limit=None, offset=None, where_document=None, include=["metadatas", "documents"] ) Key points regarding the include parameter: - Purpose: It determines which fields (besides ids, which are always returned) are included in the results [3][4]. - Available options: You can include "embeddings", "metadatas", "documents", and "uris" [1][2][5]. - Default value: If the include parameter is not specified, it defaults to ["metadatas", "documents"] [1][6]. - Optimization: To minimize response size and improve performance, it is recommended to only include the fields you explicitly need for your application [7][4]. Example usage: python # Retrieve only the documents results = collection.get(include=["documents"]) # Retrieve documents and embeddings results = collection.get(include=["documents", "embeddings"]) When using get, results are returned as a GetResult object (a dictionary), where each key corresponds to the fields requested in the include list [3][2]. These arrays are aligned by index, meaning that the record at ids[i] corresponds to the data at documents[i], metadatas[i], etc. [3][4].

Citations:


🌐 Web query:

"GetResult" "metadatas" "Optional[List[Metadata]]"

💡 Result:

The type definition metadatas: Optional[List[Metadata]] within a GetResult object is primarily associated with the ChromaDB Python API [1][2][3]. In ChromaDB, GetResult is a TypedDict representing the payload returned by collection.get() operations [1][4]. The structure typically includes the following fields, where metadatas is an optional field containing the metadata for the retrieved records [1][5]: ids: List[ID] embeddings: Optional[Union[Embeddings,...]] documents: Optional[List[Document]] uris: Optional[URIs] data: Optional[Loadable] metadatas: Optional[List[Metadata]] included: Include When using collection.get(), you can control the returned fields using the include parameter (e.g., include=["documents", "metadatas"]) [4][5]. ChromaDB returns these results in a columnar (or column-major) format, where corresponding entries across the returned lists (e.g., ids, documents, metadatas) represent the same record [1][2][4]. Other systems may use GetResult and metadata terminology, but the specific Python type hint Optional[List[Metadata]] is characteristic of ChromaDB's implementation [1][3][6][7]. For instance, FDC3 uses getResultMetadata() returning Promise<ContextMetadata> [8][9], and Oracle NoSQL uses GetResult objects where metadata is accessed via specific getter methods rather than a typed list field [6][10].

Citations:


🌐 Web query:

ChromaDB Python client includes_include_data include data metadata default True

💡 Result:

In the ChromaDB Python client, the include parameter is used to specify which optional data fields (such as "metadatas", "documents", "embeddings", "distances", or "uris") should be returned by query and get operations [1][2][3][4]. The IDs of the records are always returned by default, regardless of the include parameter [1][4]. The default behavior for the include parameter is as follows: For get: The default value is ["metadatas", "documents"] [1][2][4]. For query: The default value is ["metadatas", "documents", "distances"] [1][4]. You can override these defaults by explicitly passing an include list to the method call [3][4]. For example: collection.get(include=["documents"]) collection.query(query_texts=["example query"], include=["metadatas", "embeddings"]) There is no parameter named includes_include_data in the ChromaDB Python client API [2][5][6]. The confusion may stem from the include parameter itself, which serves the purpose of defining what data to include in the response [1][4].

Citations:


Always fetch metadata before reading it in render.

collection.get(include=['documents']) requests documents only, and GetResult.metadatas is omitted/None when not requested. results['metadatas'][i] therefore hits one of these unhandled paths: None[i] or an unrequested metadata row. Include "metadatas" in the includes list and access chunk IDs from the returned metadata dict.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_chroma/chroma.py` around lines 567 - 591, Update the
collection.get call in render to request both documents and metadatas, then read
each row’s chunkId from the returned metadata dictionary before applying the
offset and range checks. Preserve the existing text-array population logic and
avoid treating metadata as an object attribute.
nodes/src/nodes/store_elasticsearch/elasticsearch_store.py (3)

93-116: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
fd -t f 'services*.json' nodes/src/nodes/store_elasticsearch --exec cat {}
rg -nP "get\(['\"]mode['\"]" nodes/src/nodes/store_elasticsearch

Repository: rocketride-org/rocketride-server

Length of output: 761


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file sizes =="
wc -l nodes/src/nodes/store_elasticsearch/IGlobal.py nodes/src/nodes/store_elasticsearch/elasticsearch_store.py nodes/src/nodes/store_elasticsearch/configs/*.json 2>/dev/null || true

echo "== IGlobal relevant sections =="
sed -n '1,320p' nodes/src/nodes/store_elasticsearch/IGlobal.py

echo "== elasticsearch_store relevant sections =="
sed -n '1,180p' nodes/src/nodes/store_elasticsearch/elasticsearch_store.py

echo "== config files =="
fd -t f -d 3 . nodes/src/nodes/store_elasticsearch -e json -e py -e md | sort | xargs -r -n 1 printf '\n--- %s ---\n' | xargs -r -n 1 cat

echo "== mode references across node =="
rg -n "\bmode\b|get\(.*'mode'|get\(.*\"mode\"|vstore|INDEX|index" nodes/src/nodes/store_elasticsearch

Repository: rocketride-org/rocketride-server

Length of output: 20787


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate node config/schema-related files =="
git ls-files nodes/src/nodes/store_elasticsearch | sort

echo "== config/schema files =="
while IFS= read -r f; do
  if [ "${f##*.}" = "json" ] || [ "${f##*.}" = "md" ] || [ "${f##*.}" = "yml" ] || [ "${f##*.}" = "yaml" ]; then
    printf '\n--- %s ---\n' "$f"
    cat "$f"
  fi
done < <(git ls-files nodes/src/nodes/store_elasticsearch | sort)

echo "== constants =="
cat nodes/src/nodes/store_elasticsearch/constants.py

echo "== focused mode/config references =="
rg -n "\bmode\b|deploy|self-managed|vector_database|vstore|index_search|Config.getNodeConfig|getConnConfig|validation|schema|property" nodes/src/nodes/store_elasticsearch

Repository: rocketride-org/rocketride-server

Length of output: 1099


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== node files =="
git ls-files nodes/src/nodes/store_elasticsearch | sort

echo "== services and docs =="
for f in nodes/src/nodes/store_elasticsearch/*.py nodes/src/nodes/store_elasticsearch/services*.json nodes/src/nodes/store_elasticsearch/README.md nodes/src/nodes/store_elasticsearch/elasticsearch/README.md nodes/src/nodes/store_elasticsearch/opensearch/README.md nodes/src/nodes/store_elasticsearch/constants.py; do
  [ -e "$f" ] || continue
  printf '\n--- %s ---\n' "$f"
  sed -n '1,220p' "$f"
done

echo "== constants full =="
cat nodes/src/nodes/store_elasticsearch/constants.py

echo "== focused mode/config references =="
rg -n "\bmode\b|deploy|self-managed|vector_database|vstore|index_search|Config.getNodeConfig|getConnConfig|validation|schema|property" nodes/src/nodes/store_elasticsearch

Repository: rocketride-org/rocketride-server

Length of output: 50389


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== elasticsearch schema tail =="
wc -l nodes/src/nodes/store_elasticsearch/services.elasticsearch.json
sed -n '1,220p' nodes/src/nodes/store_elasticsearch/services.elasticsearch.json
sed -n '220,420p' nodes/src/nodes/store_elasticsearch/services.elasticsearch.json
sed -n '420,620p' nodes/src/nodes/store_elasticsearch/services.elasticsearch.json
sed -n '620,840p' nodes/src/nodes/store_elasticsearch/services.elasticsearch.json

echo "== README snippets =="
rg -n "mode|deploy|local|cloud|vstore|index" nodes/src/nodes/store_elasticsearch/README.md nodes/src/nodes/store_elasticsearch/elasticsearch/README.md

echo "== config schema/source evidence: whether mode is separate deployment field =="
python3 - <<'PY'
import json, pathlib
for p in pathlib.Path('nodes/src/nodes/store_elasticsearch').glob('*.json'):
    data=json.loads(p.read_text())
    print(f'--- {p}')
    props=data.get('fields',{})
    for k,v in props.items():
        if k=='elasticsearch.mode':
            print(f'ELASTICSEARCH MODE: {json.dumps(v, indent=2)[:1200]}')

# inspect service preconfig mode values that exist in git
for p in pathlib.Path('nodes/src/nodes/store_elasticsearch').glob('services*.json'):
    print(f'--- {p} preconfig mode values')
    for name,prof in data.get('preconfig',{}).get('profiles',{}).items():
        print(name, prof.get('mode'))
PY

Repository: rocketride-org/rocketride-server

Length of output: 22773


Resolve Elasticsearch store deployment mode during config mapping.

Store.__init__ derives the Elasticsearch URL scheme from config.get('mode'), but the node service defines deployment profiles with that same key and a separate elasticsearch.profile deployment field. A local profile with mode: cloud-hosted would use false store mode while constructing https://localhost:..., diverging from the documented/local scheme behavior; the store URL host scheme belongs in the deployment/profile logic, not the mode key.

🧰 Tools
🪛 ast-grep (0.44.1)

[warning] 109-109: Do not make http calls without encryption
Context: 'http://'
Note: [CWE-319] Cleartext Transmission of Sensitive Information.

(requests-http)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_elasticsearch/elasticsearch_store.py` around lines 93 -
116, Update the Elasticsearch configuration mapping before Store.__init__ so
deployment mode is resolved from the Elasticsearch deployment/profile field, not
config.mode. Pass the resolved profile-based scheme or mode into Store, and
ensure Store.__init__ uses that deployment value when selecting the default URL
scheme while preserving explicit http/https host schemes.

382-383: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Raise Exception, not BaseException.

BaseException bypasses normal except Exception handlers (and is the base of SystemExit/KeyboardInterrupt), so this can escape the pipeline's error handling unexpectedly.

🐛 Proposed fix
-        if docFilter.offset:
-            raise BaseException('Non-zero offset is not supported in semantic searching')
+        if docFilter.offset:
+            raise Exception('Non-zero offset is not supported in semantic searching')
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        if docFilter.offset:
            raise Exception('Non-zero offset is not supported in semantic searching')
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_elasticsearch/elasticsearch_store.py` around lines 382
- 383, Update the offset validation in the Elasticsearch store’s semantic-search
flow to raise Exception instead of BaseException, preserving the existing error
message and condition for non-zero docFilter.offset values.

583-591: 🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win

Payload-size batching is ineffective; sys.getsizeof(doc) only measures the dict header.

sys.getsizeof is shallow — it does not count the embedding list, content, or metadata referenced by doc, so sum_size stays tiny and sum_size > self.payload_limit essentially never fires. Only the hardcoded 500 count bound is active. Estimate the serialized size (e.g. len(json.dumps(doc['_source'])) or embedding length × item size) if the payload guard is meant to protect against large bulk requests. The literal 500 could also reuse DEFAULT_TEXT_BATCH_SIZE.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_elasticsearch/elasticsearch_store.py` around lines 583
- 591, The payload batching logic in the bulk document loop must measure
serialized document content rather than the shallow dict size from
sys.getsizeof(doc). Update the size calculation near _run_bulk to use the
serialized _source (or equivalent payload-size estimate), and reuse
DEFAULT_TEXT_BATCH_SIZE instead of the hardcoded 500 when appropriate; preserve
flushing when either count or payload limits are reached.
nodes/src/nodes/store_elasticsearch/README.md (3)

37-75: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Fix the Elasticsearch configuration table.

This section mixes OpenSearch fields (host, enabled, username, password, collection) and even lists provider as "opensearch" under the Elasticsearch heading. That contradicts the generated services.elasticsearch.json schema and the Elasticsearch loader, which use profile/vector fields and provider: "elasticsearch". Users following this table will configure the wrong keys.

🧰 Tools
🪛 LanguageTool

[style] ~50-~50: Consider removing “of” to be more concise
Context: ...rms, 'and' matches documents containing ALL of the query terms, 'exact' requires the exact...

(ALL_OF_THE)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_elasticsearch/README.md` around lines 37 - 75, Replace
the outdated Elasticsearch configuration table with entries matching the
services.elasticsearch.json schema and Elasticsearch loader: use the
profile/vector fields shown in the lower table, remove OpenSearch-only keys such
as host, enabled, username, password, and collection, and set provider to
"elasticsearch" where applicable. Keep the documented defaults and
profile-specific fields accurate.

124-126: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate files =="
fd -a 'README.md|store_elasticsearch|opensearch|elasticsearch|node' nodes . | sed 's#^\./##' | head -100

echo "== target README section =="
if [ -f nodes/src/nodes/store_elasticsearch/README.md ]; then
  nl -ba nodes/src/nodes/store_elasticsearch/README.md | sed -n '1,220p'
fi

echo "== search references =="
rg -n "verify_certs|opensearch|OpenSearch|elastic|store_elasticsearch|http://.*https|self-signed|certificate" nodes/src/nodes nodes -S --glob '!**/*.map' | head -200

Repository: rocketride-org/rocketride-server

Length of output: 5986


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== target README =="
if [ -f nodes/src/nodes/store_elasticsearch/README.md ]; then
  nl -ba nodes/src/nodes/store_elasticsearch/README.md | sed -n '1,220p'
else
  echo "missing target README"
fi

echo "== store_elasticsearch files =="
find nodes/src/nodes/store_elasticsearch -maxdepth 4 -type f -print | sort

echo "== OpenSearch/verify_certs/elasticsearch references =="
rg -n -C 3 "verify_certs|opensearch|OpenSearch|http://.*https|self-signed|certificate|auth" nodes/src/nodes/store_elasticsearch nodes/src/nodes -S | head -250

Repository: rocketride-org/rocketride-server

Length of output: 230


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== target README =="
if [ -f nodes/src/nodes/store_elasticsearch/README.md ]; then
  awk '{printf "%6d\t%s\n", NR, $0}' nodes/src/nodes/store_elasticsearch/README.md | sed -n '1,220p'
else
  echo "missing target README"
fi

echo "== store_elasticsearch files =="
find nodes/src/nodes/store_elasticsearch -maxdepth 4 -type f -print | sort

echo "== OpenSearch/verify_certs/elasticsearch references =="
rg -n -C 3 "verify_certs|opensearch|OpenSearch|http://.*https|self-signed|certificate|auth" nodes/src/nodes/store_elasticsearch nodes/src/nodes -S | head -300

Repository: rocketride-org/rocketride-server

Length of output: 47710


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== opensearch client outline =="
if [ -f nodes/src/nodes/store_elasticsearch/opensearch_client.py ]; then
  awk '{printf "%6d\t%s\n", NR, $0}' nodes/src/nodes/store_elasticsearch/opensearch_client.py | sed -n '1,240p'
fi

echo "== IGlobal init relevant sections =="
if [ -f nodes/src/nodes/store_elasticsearch/IGlobal.py ]; then
  awk '{printf "%6d\t%s\n", NR, $0}' nodes/src/nodes/store_elasticsearch/IGlobal.py
fi

Repository: rocketride-org/rocketride-server

Length of output: 28980


Do not describe disabled TLS verification as suitable for production.

IGlobal._begin_opensearch() initializes OpenSearch with verify_certs=False and ssl_show_warn=False; when basic auth is enabled, credentials are sent over a connection whose server certificate is not validated, so this is an insecure compatibility fallback, not a safe default. Update the OpenSearch docs to make the insecure fallback explicit and add/configure a CA verification option before using production deployments with self-signed certificates.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_elasticsearch/README.md` around lines 124 - 126, Update
the OpenSearch “Basic auth” documentation to describe verify_certs=False and
ssl_show_warn=False as an insecure compatibility fallback, not suitable for
production. Explicitly instruct production users with self-signed certificates
to configure CA certificate verification before deployment, while preserving the
existing authentication requirements.

187-187: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Regenerate the source link after the package move.

The generated README still links to the old nodes/src/nodes/index_search path, while the package and service path are now under nodes/src/nodes/store_elasticsearch. Update the generator/source metadata and rerun nodes:docs-generate; do not hand-edit the generated block.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_elasticsearch/README.md` at line 187, Update the source
metadata used by the README generator for the Elasticsearch node so its
generated source link targets nodes/src/nodes/store_elasticsearch instead of
nodes/src/nodes/index_search. Then rerun nodes:docs-generate to regenerate the
README block; do not edit the generated link manually.

Source: Coding guidelines

nodes/src/nodes/store_milvus/milvus.py (7)

104-109: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== files =="
git ls-files | rg '(^|/)milvus\.py$|(^|/)ROCKETRIDE_python_API\.md$|config\.json$' || true

echo "== target snippet and surrounding relevant lines =="
sed -n '1,240p' nodes/src/nodes/store_milvus/milvus.py | cat -n

echo "== searches for similarity/config usage =="
rg -n "similarity|FLOAT_VECTOR|create_collection|insert_index|index" nodes/src/nodes/store_milvus nodes -g '*.py' -g '*.json' || true

Repository: rocketride-org/rocketride-server

Length of output: 50389


🌐 Web query:

Milvus documentation FLOAT_VECTOR index metric_type L2 IP COSINE JACCARD HAMMING BM25

💡 Result:

In Milvus, supported metric types depend on the vector field data type [1]. For fields of type FLOAT_VECTOR, the supported similarity metrics are COSINE, L2, and IP [1][2]. The metric types you mentioned are categorized based on the vector field type as follows: FLOAT_VECTOR (and other dense floating-point vector types like FLOAT16_VECTOR, BFLOAT16_VECTOR, INT8_VECTOR): Supported metrics are COSINE, L2, and IP [1]. BINARY_VECTOR: Supported metrics are JACCARD and HAMMING [1][3]. SPARSE_FLOAT_VECTOR: Supported metrics are IP and BM25 [1]. Note that BM25 is specifically designed for full-text search and is used exclusively with sparse vectors (SPARSE_FLOAT_VECTOR), not with FLOAT_VECTOR [1][4][5]. When using BM25, the required index type is typically SPARSE_INVERTED_INDEX [6][4].

Citations:


Restrict Milvus float-vector similarity metrics to supported values.

vector is a FLOAT_VECTOR, but config values JACCARD, HAMMING, and BM25 pass this validation even though MILVUS only supports COSINE, L2, and IP for float vectors (JACCARD/HAMMING are for binary vectors; BM25 is for sparse vectors). These values would be accepted and then fail collection/index creation because they are invalid for this schema.

🐛 Proposed fix
-        if similarity in ['L2', 'IP', 'COSINE', 'JACCARD', 'HAMMING', 'BM25']:
+        if similarity in ['L2', 'IP', 'COSINE']:
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        # check if the similarity matches milvus configuration options
        similarity = config.get('similarity', 'COSINE')
        if similarity in ['L2', 'IP', 'COSINE']:
            self.similarity = similarity
        else:
            raise Exception('The metric you provided in the config.json does not match required milvus configurations')
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_milvus/milvus.py` around lines 104 - 109, Update the
similarity validation in the Milvus initialization logic to accept only COSINE,
L2, and IP for the FLOAT_VECTOR schema. Remove JACCARD, HAMMING, and BM25 from
the allowed values while preserving the existing exception for unsupported
metrics.

205-207: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate file =="
fd -a 'milvus.py$' . || true

echo "== file excerpt around relevant lines =="
wc -l nodes/src/nodes/store_milvus/milvus.py
sed -n '1,260p' nodes/src/nodes/store_milvus/milvus.py | cat -n

echo "== search for docFilter isTable stores =="
rg -n "isTable|isDeleted|meta\[" nodes/src/nodes/store_milvus/milvus.py . --glob '*.py' || true

Repository: rocketride-org/rocketride-server

Length of output: 49987


🌐 Web query:

Milvus JSON boolean filter BooleanExpression isTable == bool string literal

💡 Result:

In Milvus, to filter on a boolean value within a JSON field, you use the standard comparison operator == with a boolean literal [1]. The boolean literal should be written as true or false (without quotes) [1]. If you have a JSON field named metadata and want to filter where the key isTable is true, your filter expression should look like this: filter = 'metadata["isTable"] == true' Key points for implementation: - Boolean Literals: Use true or false directly. Do not wrap them in quotes, as wrapping them in quotes would treat them as string literals (e.g., "true") rather than boolean values [2][1]. - JSON Path Syntax: Use bracket notation (e.g., field["key"]) to access keys within a JSON field [1][3]. - Indexing: For optimal performance, ensure that if you have created an index on the JSON field, the json_cast_type is set to bool [1]. Filtering with a different type than the indexed json_cast_type may cause the system to bypass the index and perform a slower brute-force scan [1].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== local Milvus filter tests around boolean expressions =="
sed -n '1,260p' nodes/test/milvus/test_filter_injection.py | cat -n
rg -n "isTable|bool|false|false|True|False" nodes/test/milvus nodes/src/nodes/store_milvus --glob '*.py' || true

echo "== Milvus filter behavior probe from mock expression matching =="
sed -n '260,720p' nodes/test/mocks/pymilvus/__init__.py | cat -n

Repository: rocketride-org/rocketride-server

Length of output: 27397


Use unquoted boolean literals for Milvus JSON boolean filters.

_escape_milvus_str(docFilter.isTable) converts True / False to 'True' / 'False' and wraps them in quotes, producing malformed Milvus filter expressions. Compare meta['isTable'] with the lowercase boolean literals used by Milvus, e.g. meta['isTable'] == true / meta['isTable'] == false, or build the boolean filter separately. The current (...,) wrapper also makes append() return a tuple; drop it.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_milvus/milvus.py` around lines 205 - 207, Update the
docFilter.isTable condition to append a Milvus filter using lowercase, unquoted
boolean literals derived from the boolean value, rather than _escape_milvus_str.
Remove the extraneous tuple wrapper around must_conditions.append so the
condition is appended directly.

417-431: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Inconsistent escaping mechanism: json.dumps() here vs. _escape_milvus_str() everywhere else.

getPaths() (line 431) and the delete filter in addChunks() (line 473) build filter literals with json.dumps(), while every other filter-construction site in this file uses the dedicated, unit-tested _escape_milvus_str() helper (see test_filter_injection.py). Both approaches likely produce safe output, but consolidating onto one tested escaping path reduces the chance of a future edge case slipping through only one of the two mechanisms.

🧰 Tools
🪛 ast-grep (0.44.1)

[info] 430-430: use jsonify instead of json.dumps for JSON output
Context: json.dumps(parent)
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_milvus/milvus.py` around lines 417 - 431, Replace the
json.dumps-based escaping in getPaths and the delete filter within addChunks
with the existing _escape_milvus_str helper. Preserve the current filter
semantics while routing both parent values through the shared, tested escaping
mechanism.

450-520: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win

Existing chunks are deleted before validating that all new chunks have embeddings — data loss on partial failure.

addChunks deletes all existing records for the incoming objectIds (lines 472-478) before the per-chunk loop that validates embedding is not None (lines 500-502) and performs the upsert. If any chunk in the batch lacks an embedding, the exception aborts the method after the old data has already been deleted but before any (or all) new chunks are inserted, permanently losing the previous version of the document with no rollback.

Move the embedding validation ahead of the delete so a bad chunk aborts before any destructive change is made.

🧰 Tools
🪛 ast-grep (0.44.1)

[info] 472-472: use jsonify instead of json.dumps for JSON output
Context: json.dumps(k)
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)


[info] 505-505: use secrets package over random package
Context: random.getrandbits(27)
Note: [CWE-330] Use of Insufficiently Random Values.

(avoid-random-python)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_milvus/milvus.py` around lines 450 - 520, In addChunks,
validate that every chunk has a non-None embedding before building objectIds or
executing the existing deletion via self.client.delete. Move the embedding check
out of the insertion loop into a preliminary validation pass, while preserving
the current exception behavior and upsert flow for valid batches.

663-666: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

render() reads point['chunkId'] directly, but the query only requests meta and content — this will KeyError.

The query at line 652-654 uses output_fields=['meta', 'content']; chunkId is nested inside the meta JSON blob, not a top-level key (as correctly handled in getPaths() via record.get('meta', {}) and _convertToDocs() via entity.get('meta')). As written, render() will raise KeyError: 'chunkId' on the first non-empty result, breaking document rendering entirely for this store.

🐛 Proposed fix
             for point in results:
                 content = point['content']
-                chunk = point['chunkId']
+                chunk = point['meta']['chunkId']
                 index = chunk - offset
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

            for point in results:
                content = point['content']
                chunk = point['meta']['chunkId']
                index = chunk - offset
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_milvus/milvus.py` around lines 663 - 666, Update
render() to read chunkId from the result’s meta JSON/object, matching the
existing getPaths() and _convertToDocs() handling, instead of accessing
point['chunkId'] directly. Preserve the current chunk-offset calculation and
rendering behavior after extracting the nested value.

144-193: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Files:"
git ls-files | rg '(^|/)milvus\.py$|test_addchunks_return\.py$|Store.*\.py$|DocumentStore' | head -200

echo
echo "milvus outline:"
ast-grep outline nodes/src/nodes/store_milvus/milvus.py --match Milvus --view expanded || true

echo
echo "Relevant milvus lines 80-220:"
sed -n '80,220p' nodes/src/nodes/store_milvus/milvus.py | nl -ba -v80

echo
echo "Relevant milvus lines 440-470:"
sed -n '440,470p' nodes/src/nodes/store_milvus/milvus.py | nl -ba -v440

echo
echo "Search createCollection definitions/usages:"
rg -n "def createCollection|def _createCollection|_createCollection|createCollection\\(" nodes/src -S

Repository: rocketride-org/rocketride-server

Length of output: 401


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Relevant milvus lines 80-220:"
sed -n '80,220p' nodes/src/nodes/store_milvus/milvus.py | awk '{printf "%4d\t%s\n", NR+79, $0}'

echo
echo "Relevant milvus lines 440-470:"
sed -n '440,470p' nodes/src/nodes/store_milvus/milvus.py | awk '{printf "%4d\t%s\n", NR+439, $0}'

echo
echo "Search createCollection definitions/usages:"
rg -n "def createCollection|def _createCollection|_createCollection|createCollection\\(" nodes/src -S

echo
echo "Test addchunks return relevant lines:"
if [ -f nodes/test/milvus/test_addchunks_return.py ]; then
  awk '{printf "%4d\t%s\n", NR, $0}' nodes/test/milvus/test_addchunks_return.py
fi

echo
echo "Find DocumentStore parent files:"
rg -n "class .*Milvus|DocumentStore|createCollection\b" nodes/src/nodes/store_milvus/milvus.py -S

Repository: rocketride-org/rocketride-server

Length of output: 11777


🌐 Web query:

Milvus Python client DataType max_length VARCHAR primary key max_length documentation

💡 Result:

When using the Milvus Python client (pymilvus), a VARCHAR field used as a primary key must have a defined max_length property [1][2]. The value of max_length defines the maximum byte length for strings allowed to be inserted into that field [1][2]. Key details regarding VARCHAR max_length in Milvus: 1. Valid Range: The permitted value range for max_length is [1] [1][2][3]. 2. Mandatory Requirement: Setting max_length is mandatory when defining a field with DataType.VARCHAR [1][2][3]. 3. Byte vs. Character Length: The max_length parameter specifies the maximum number of bytes, not the number of characters [1][2]. Because multibyte characters (such as certain Unicode/UTF-8 characters) may occupy more than one byte each, you should ensure that the byte length of your strings does not exceed the configured limit [1][2]. 4. Immutability: Once a collection is created, the max_length of a VARCHAR field cannot be changed [4]. 5. Internal Logic: The max_length value is used by Milvus for internal operations, such as allocating memory for insert buffers, so it is recommended to choose an accurate value rather than always defaulting to the maximum unless necessary [4]. Example of defining a VARCHAR primary key in your schema: from pymilvus import DataType schema.add_field( field_name="your_pk_field", datatype=DataType.VARCHAR, is_primary=True, auto_id=False, # Set to True if you want Milvus to generate IDs instead max_length=100 # Required for VARCHAR; range [1])

Citations:


Correct _createCollection’s success/failure return contract.

_createCollection returns False when a new collection is created successfully, but addChunks() checks not self.createCollection(chunks) and skips all writes on False; creation exceptions are currently reported as True (success). Adjust it so successful collection creation/existence returns True and creation failures return False.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_milvus/milvus.py` around lines 144 - 193, The
_createCollection method currently reverses the expected success contract.
Update its return values so an existing collection and successful
create_collection call return True, while exceptions during creation return
False; preserve the existing collection-existence check and deadlock-avoidance
behavior.

156-156: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== repo files matching milvus.py =="
fd -a 'milvus.py$' . | sed 's#^\./##'

echo "== target excerpt =="
cat -n nodes/src/nodes/store_milvus/milvus.py | sed -n '130,175p'

echo "== pymilvus usages and imports =="
rg -n "from pymilvus|import pymilvus|max_length|DataType\.INT64|schema\.add_field" -S .

echo "== python/pymilvus availability =="
python3 - <<'PY'
try:
    import pymilvus
    print("pymilvus", getattr(pymilvus, "__version__", "unknown"))
    from pymilvus import DataType
    print("DataType.INT64", DataType.INT64)
    print("DataType.VARCHAR", DataType.VARCHAR)
except Exception as e:
    print("pymilvus unavailable:", type(e).__name__, str(e))
PY

Repository: rocketride-org/rocketride-server

Length of output: 7746


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== pyproject/package metadata references =="
rg -n 'pymilvus|MilvusClient|store_milvus|nodes/src/nodes/store_milvus' -S pyproject.toml requirements*.txt setup.py setup.cfg poetry.lock uv.lock packages manifest* nodes 2>/dev/null | head -200

echo "== tests/mock schema behavior =="
cat -n nodes/src/nodes/store_milvus/milvus.py | sed -n '1,220p'
echo
cat -n test/nodes/connectors/milvus/store/test_milvus.py | sed -n '1,140p'
echo
cat -n nodes/test/mocks/pymilvus/__init__.py | sed -n '120,190p'

Repository: rocketride-org/rocketride-server

Length of output: 5571


🌐 Web query:

pymilvus MilvusClient create_schema add_field max_length DataType.INT64 max_length ignored mandatory VARCHAR documentation

💡 Result:

In the PyMilvus SDK, the max_length parameter is strictly reserved for string-based data types, specifically DataType.VARCHAR [1][2][3]. When adding a field to a collection schema using add_field, max_length is mandatory for DataType.VARCHAR fields to define the maximum byte length of strings (valid range: 1 to 65,535) [1][2][4][3]. For other data types, such as DataType.INT64, the max_length parameter is not applicable [1][2]. If provided, it is typically ignored or considered invalid for non-string fields [1][2]. Key takeaways regarding your query: 1. Mandatory vs. Ignored: The max_length parameter is mandatory for VARCHAR fields [1][2][5]. If you attempt to use it with DataType.INT64, it serves no function and is ignored [1][2]. 2. Documentation: Official documentation consistently specifies that max_length is required for VARCHAR [1][2][3]. It does not list max_length as a valid parameter for numeric types like INT64 [1][2][6]. 3. Best Practice: To avoid confusion or potential schema validation warnings, you should omit the max_length parameter entirely when defining non-string fields such as INT64 [1][2]. For further details, refer to the official PyMilvus API reference for CollectionSchema.add_field [1][2] or the general schema management documentation [6][5].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== constraints.lock pymilvus entries =="
sed -n '940,1235p' nodes/src/nodes/constraints.lock | rg -n 'pymilvus|Package|Version|Metadata|pydantic|protobuf|grpc|milvus' -C 2 || true

echo "== pyproject constraints files =="
fd -a 'pyproject.toml|constraints.lock|requirements.*|setup.*|poetry.lock|uv.lock' nodes/src/nodes | sed 's#^\./##'

Repository: rocketride-org/rocketride-server

Length of output: 9302


🌐 Web query:

pymilvus 3.0.0 source CollectionSchema add_field max_length DataType.INT64 validation

💡 Result:

In PyMilvus 3.0.0, the max_length parameter is not applicable to DataType.INT64 fields. The max_length parameter is specifically mandatory and exclusively used for DataType.VARCHAR fields to define the maximum byte length of string data [1][2][3]. For DataType.INT64 fields: - They are scalar fields used for storing 64-bit integers [4]. - They do not support max_length because the size of an integer is fixed by the data type itself [4]. - If you are configuring a field with DataType.INT64, you should not include the max_length parameter, as it is intended for string-based validation [1][2]. Validation for max_length is strictly enforced for VARCHAR fields, with a valid range of [1] bytes [1][3]. Attempting to apply max_length to an INT64 field would be technically incorrect according to the Milvus schema design [1][2].

Citations:


🌐 Web query:

"MilvusClient.create_schema" "add_field" "max_length" "DataType.INT64"

💡 Result:

In Milvus, the max_length parameter is not used with DataType.INT64 [1][2][3]. The max_length parameter is specifically used for DataType.VARCHAR fields to define the maximum number of bytes (ranging from 1 to 65,535) that the string field can store [4][5][6]. For DataType.INT64 fields, Milvus handles the storage automatically as a 64-bit integer, and no length or dimension parameters are required [2][3]. If you are adding an INT64 field to your schema, you should omit max_length [5][7]. Example of correctly adding an INT64 field: schema.add_field(field_name="my_int64_field", datatype=DataType.INT64) [2][5]

Citations:


🌐 Web query:

site:github.com Milvus-ORM pymilvus add_field max_length DataType.INT64

💡 Result:

In Milvus and the Pymilvus ORM, the max_length parameter is not applicable to DataType.INT64 [1]. The max_length property is specifically used for DataType.VARCHAR fields to define the maximum character length of the string [2][1]. DataType.INT64 fields in Milvus are 64-bit integers and have a fixed size; they do not require or support length-related parameters [3]. When defining a schema using add_field or FieldSchema, you should only provide max_length when the data type is DataType.VARCHAR [1][4]. Providing max_length for an INT64 field is not a supported configuration [1].

Citations:


Remove the irrelevant max_length from the INT64 schema field.

max_length applies to VARCHAR fields; on the DataType.INT64 primary key this is misleading configuration and should be omitted.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_milvus/milvus.py` at line 156, Remove the max_length
argument from the INT64 primary-key field definition in schema.add_field, while
preserving the existing field_name, datatype, and is_primary settings.
nodes/src/nodes/store_pinecone/pinecone.py (3)

68-68: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Score threshold never applied in searchSemantic — attribute name typo.

threshhold_search (line 68, extra “h”) is never assigned; __init__ sets self.threshold_search (line 90, correct spelling) instead. Line 290 filters against self.threshhold_search, which stays at its class-level default 0.0 forever, so the configured score threshold is silently ignored for semantic search — every match passes the filter regardless of similarity.

🐛 Proposed fix
-    threshhold_search: float = 0.0
+    threshold_search: float = 0.0
@@
-        points_satisfy_threshold = [point for point in points if point['score'] >= self.threshhold_search]
+        points_satisfy_threshold = [point for point in points if point['score'] >= self.threshold_search]

Also applies to: 90-90, 289-290

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_pinecone/pinecone.py` at line 68, Correct the
misspelled class attribute threshhold_search to threshold_search and update the
searchSemantic filtering reference accordingly, matching the
self.threshold_search assignment in __init__. Ensure semantic search uses the
configured score threshold instead of the class-level default.

226-249: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

searchKeyword crashes: passes a raw Pinecone filter dict into get(), which re-converts it as a DocFilter.

searchKeyword builds filter via _convertFilter(docFilter) (already a Pinecone {'$and': [...]} dict), appends the $contains clause, then calls self.get(filter). But get()'s parameter is typed/used as a DocFilter and immediately does self._convertFilter(docFilter=docFilter) — which accesses docFilter.isDeleted, .nodeId, etc. as attributes. Since a plain dict has none of these, this raises AttributeError on every call to searchKeyword.

🐛 Proposed fix: split filter-application from filter-conversion
-    def get(self, docFilter: DocFilter, checkCollection: bool = True) -> List[Doc]:
-        """
-        Given a filter, return the document groups matching the filter.
-        """
-        # If the collection does not exists, by definition there are
-        # no documents matching the get
-        if checkCollection and not self.doesCollectionExist():
-            return []
-
-        # Build up the filter
-        filter = self._convertFilter(docFilter=docFilter)
-
-        # Getting index
-        index = self.client.Index(self.collection)
-        vector_size = int(index.describe_index_stats()['dimension'])
-
-        # Perform the search
-        records = index.query(
-            vector=[1] * vector_size, top_k=10000, filter=filter, include_metadata=True, include_values=False
-        )['matches']
-
-        # Convert the points into documents
-        docs = self._convertToDocs(records)
-        return docs
+    def get(self, docFilter: DocFilter, checkCollection: bool = True) -> List[Doc]:
+        """
+        Given a filter, return the document groups matching the filter.
+        """
+        if checkCollection and not self.doesCollectionExist():
+            return []
+        filter = self._convertFilter(docFilter=docFilter)
+        return self._queryFilter(filter)
+
+    def _queryFilter(self, filter: dict) -> List[Doc]:
+        """Run a pre-built Pinecone filter and convert matches to Docs."""
+        index = self.client.Index(self.collection)
+        vector_size = int(index.describe_index_stats()['dimension'])
+        records = index.query(
+            vector=[1] * vector_size, top_k=10000, filter=filter, include_metadata=True, include_values=False
+        )['matches']
+        return self._convertToDocs(records)
-        # Convert the points into groups
-        docs = self.get(filter)
-
-        # Return them
-        return docs
+        # Convert the points into groups
+        return self._queryFilter(filter)

Also applies to: 297-320

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_pinecone/pinecone.py` around lines 226 - 249, Update
searchKeyword and the related get flow so the already-converted Pinecone filter
is not passed back through _convertFilter as a DocFilter. Separate filter
conversion from filter application, or otherwise reuse the converted filter
directly, while preserving the existing DocFilter conversion behavior for normal
get callers.

399-468: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

addChunks deletes all pre-existing chunks for an object before all new batches finish upserting.

objectIds is collected once for the entire chunks argument (lines 424-426), but self.remove(list(objectIds.keys())) only runs inside flush() on whichever call happens to be first (after which objectIds is cleared). This means the old data for a document is fully deleted before later batches of the same re-ingest have been written. If the pipeline fails/crashes between the first and last flush(), the old chunks are gone and only a partial set of new chunks exists — permanent data loss for that objectId.

Since Pinecone has no cross-batch transaction, upsert all new chunks first and only delete stale ones once every batch has succeeded, so a mid-way failure leaves both old and new data present (self-healing on retry) instead of losing data.

🛡️ Proposed fix: defer deletion until after all upserts complete
     def flush():
         nonlocal points_to_upsert
-        nonlocal objectIds
-
-        # Getting index
         index = self.client.Index(self.collection)
-
-        # Batch operation for deletion
-        if len(objectIds):
-            self.remove(list(objectIds.keys()))
-
-        # Batch operation for upsert
         if len(points_to_upsert):
             index.upsert(vectors=points_to_upsert)
-
-        # Clear them
-        objectIds = {}
         points_to_upsert = []
@@
-    flush()
+    flush()
+    if len(objectIds):
+        self.remove(list(objectIds.keys()))
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        points_to_upsert = []

        # Clear the object id list
        objectIds: Dict = {}

        def flush():
            nonlocal points_to_upsert

            # Getting index
            index = self.client.Index(self.collection)

            # Batch operation for upsert
            if len(points_to_upsert):
                index.upsert(vectors=points_to_upsert)

            # Clear them
            points_to_upsert = []

        # For each document set objectId to true to make sure we delete any chunks we already have added
        for chunk in chunks:
            if chunk.metadata.objectId != 'schema':
                objectIds[chunk.metadata.objectId] = True

        sum_size = 0
        cur_size = 0
        # For each document
        for chunk in chunks:
            # Get the embedding
            embedding = chunk.embedding

            # If we do not have an embedding
            if embedding is None:
                raise Exception('No embedding in document')

            metadata = dict(chunk.metadata)
            metadata['content'] = chunk.page_content

            # Bogus document which is not compatible with pinecone because it can't accept all zeroes
            if chunk.metadata.objectId == 'schema':
                embedding[-1] = 1
                metadata['nodeId'] = ''
                metadata['parent'] = ''
                metadata['permissionId'] = 0
                metadata['isTable'] = False
                metadata['tableId'] = 0

            # Append the points // create a unique identifier that fits into an int64 id field
            vector_struct = {
                'id': str(uuid4()),
                'values': embedding,
                'metadata': metadata,
            }

            cur_size = sys.getsizeof(vector_struct)
            sum_size += cur_size
            points_to_upsert.append(vector_struct)

            # Pinecone maxes out at 100 for bulk upsert but chose 50 to ensure it doesn't hit the limit
            if (len(points_to_upsert) >= 50) or (sum_size + cur_size > self.payload_limit):
                flush()
                cur_size = 0
                sum_size = 0

        flush()
        if len(objectIds):
            self.remove(list(objectIds.keys()))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_pinecone/pinecone.py` around lines 399 - 468, Update
addChunks so flush only upserts pending points and does not call self.remove or
clear objectIds between batches. After the loop’s final flush succeeds, delete
the collected object IDs once, ensuring all new chunks are written before stale
data is removed and failures leave existing data intact.
nodes/src/nodes/store_pinecone/README.md (1)

52-59: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Profiles section default contradicts the Fields table and generated Schema.

This section states The default profile is serverless-dense and marks serverless-dense as yes/default, but the Fields table (line 36) and the auto-generated Schema section (line 112) both say the actual default is "pod-based". Update the Profiles table/prose to match the real default so the docs don't mislead users about which deployment gets created when profile is unset.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_pinecone/README.md` around lines 52 - 59, Update the
Profiles table and its concluding default-profile statement in the README to use
pod-based as the default, matching the Fields table and generated Schema; mark
serverless-dense as non-default and pod-based as default without changing their
deployment descriptions.
nodes/src/nodes/store_postgres/postgres.py (3)

123-127: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add a connect timeout to the production DB connection.

IGlobal.validateConfig() explicitly uses connect_timeout=3 for its probe connection, but this constructor — the connection actually used for all reads/writes — has none. If the host is unreachable or the network stalls, psycopg2.connect() can hang indefinitely, blocking beginGlobal()/pipeline startup on this thread.

🔌 Proposed fix
         self.client = psycopg2.connect(
-            dbname=self.database, user=self.user, password=self.password, host=self.host, port=self.port
+            dbname=self.database, user=self.user, password=self.password, host=self.host, port=self.port,
+            connect_timeout=10,
         )
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

        self.client = psycopg2.connect(
            dbname=self.database, user=self.user, password=self.password, host=self.host, port=self.port,
            connect_timeout=10,
        )

        register_vector(self.client)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_postgres/postgres.py` around lines 123 - 127, Update
the production connection in the store-postgres constructor to pass psycopg2’s
connect_timeout parameter, matching the 3-second timeout used by
IGlobal.validateConfig(). Keep the existing connection arguments and
register_vector(self.client) behavior unchanged.

408-478: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

No rollback on failed statements — a single error poisons the shared connection for the rest of the Store's lifetime.

self.client is a single persistent psycopg2 connection reused across _createCollection, addChunks, remove, markDeleted, markActive, and every search method. None of these ever call self.client.rollback(). If any statement in addChunks's per-chunk insert loop (lines 424-439) raises (bad embedding size, unique/constraint violation, transient network error), Postgres puts the transaction into the aborted state; every subsequent query on this connection then fails with "current transaction is aborted" until the process is restarted — including unrelated reads for other objects. This is a real reliability risk for a long-lived store used across many pipeline instances.

Wrap write paths in try/except and roll back on failure, e.g.:

try:
    ...
    self.client.commit()
except Exception:
    self.client.rollback()
    raise

Apply this consistently to _createCollection, addChunks, remove, markDeleted, and markActive.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_postgres/postgres.py` around lines 408 - 478, Update
the write methods _createCollection, addChunks, remove, markDeleted, and
markActive to wrap their cursor operations and commit in try/except blocks; on
any exception, call self.client.rollback() before re-raising the original
exception. Preserve existing behavior on successful commits and early returns.

424-439: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

Batch the chunk inserts instead of one INSERT per chunk.

The loop executes a separate round-trip INSERT for every chunk. For large ingests this is an unnecessary N-query hot path; consider psycopg2.extras.execute_values (or building one multi-row INSERT) to insert all chunks in a single statement.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_postgres/postgres.py` around lines 424 - 439, The chunk
insertion flow using self.client.cursor and SQL_QUERIES['insert_chunk'] should
batch all chunks into one database statement instead of executing one INSERT per
iteration. Build the complete row values and shared column list first, then use
psycopg2.extras.execute_values or an equivalent multi-row INSERT, preserving
vectorSize and modelName defaults before committing once.
nodes/test/conftest.py (1)

320-321: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Temporary skip — ensure it gets tracked for re-enablement.

store_elasticsearch is placed under the "exclude nodes with failing tests until they can be fixed" group, so its dynamically generated tests won't run in default CI. This matches the rename and the #1642 dependency noted in the PR, but a lingering skip can mask regressions. Want me to open a follow-up issue to track re-enabling these tests once #1642 lands?

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/test/conftest.py` around lines 320 - 321, Track the temporary exclusion
of store_elasticsearch for re-enablement by adding a follow-up issue reference
or TODO next to its entry in the failing-test exclusion group. Keep the
exclusion unchanged until dependency `#1642` is resolved, and make the tracking
explicit so the dynamically generated tests can be restored afterward.
packages/docs/redirects.ts (1)

183-197: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Preserve Elasticsearch/OpenSearch variant deep links.

The provider-specific sources in Lines 187-195 currently redirect to the generic /nodes/store_elasticsearch overview, even though the documentation has nested Elasticsearch and OpenSearch pages. Keep /nodes/index_search mapped to the overview, but route the provider-specific legacy paths to their corresponding nested pages.

Proposed redirect split
   {
     "to": "/nodes/store_elasticsearch",
     "from": [
-      "/nodes/index_search",
-      "/store/elasticsearch",
-      "/store/elasticsearch/elasticsearch"
+      "/nodes/index_search"
+    ]
+  },
+  {
+    "to": "/nodes/store_elasticsearch/elasticsearch",
+    "from": [
+      "/store/elasticsearch",
+      "/store/elasticsearch/elasticsearch"
     ]
   },
   {
-    "to": "/nodes/store_elasticsearch",
+    "to": "/nodes/store_elasticsearch/opensearch",
     "from": [
       "/store/opensearch",
       "/store/opensearch/opensearch"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

  {
    "to": "/nodes/store_elasticsearch",
    "from": [
      "/nodes/index_search"
    ]
  },
  {
    "to": "/nodes/store_elasticsearch/elasticsearch",
    "from": [
      "/store/elasticsearch",
      "/store/elasticsearch/elasticsearch"
    ]
  },
  {
    "to": "/nodes/store_elasticsearch/opensearch",
    "from": [
      "/store/opensearch",
      "/store/opensearch/opensearch"
    ]
  },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/docs/redirects.ts` around lines 183 - 197, Update the redirect
entries in the redirects configuration so /nodes/index_search continues
targeting /nodes/store_elasticsearch, while Elasticsearch-specific legacy paths
target the nested Elasticsearch page and OpenSearch-specific paths target the
nested OpenSearch page. Preserve the existing source paths and split the
provider-specific mappings accordingly.
nodes/src/nodes/store_milvus/IGlobal.py (1)

74-77: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file outline =="
ast-grep outline nodes/src/nodes/store_milvus/IGlobal.py --view expanded || true

echo "== relevant IGlobal.py lines =="
sed -n '1,130p' nodes/src/nodes/store_milvus/IGlobal.py | cat -n

echo "== locate milvus.py and runtime URI handling =="
fd -a 'milvus\.py$' . | sed "s#^\./##" | while read -r f; do
  echo "--- $f ---"
  rg -n "connections\.connect|uri\s*=|https://" "$f" -C 3 || true
done

echo "== python imports / dependencies =="
git ls-files | rg '(^|/)(pyproject\.toml|requirements.*\.txt|environment.*\.ya?ml|Pipfile)$' || true
rg -n "pymilvus|milvus" -g 'pyproject.toml' -g 'requirements*.txt' -g '*.py' . | head -80 || true

Repository: rocketride-org/rocketride-server

Length of output: 22160


🌐 Web query:

pymilvus connections.connect Illegal uri expected form http[s]://

💡 Result:

The error message "Illegal uri:..., expected form 'http[s]://[user:password@]example.com[:12345]'" in PyMilvus occurs because the uri parameter in connections.connect requires a properly formatted URI string [1][2]. To resolve this, ensure your uri string includes the protocol (http or https) [1][3]. Instead of passing just the host and port (e.g., "localhost:19530"), use the full URI format [1][4]: connections.connect( alias="default", uri="http://localhost:19530") Alternatively, you can provide the host and port as separate arguments, which does not require the URI string format [4][5]: connections.connect( alias="default", host="localhost", port="19530") If you are encountering this error while trying to connect to Milvus Lite (a local file-based instance), it may be because you are using an outdated version of PyMilvus [6][7]. Support for Milvus Lite was introduced in later versions; ensure your installation is at least version 2.4.2 or higher [6][7]: pip install pymilvus>=2.4.2

Citations:


Use a scheme for the cloud validation URI.

connections.connect(uri=..., token=...) expects http[s]://host[:port]; a bare host:port raises Illegal uri. The runtime Milvus client path adds https:// for cloud configs, so this save-time probe should use the same URI shape instead of failing valid token configurations.

🐛 Proposed fix
                 if token:
                     # Cloud: pass UI host and port verbatim via URI
-                    uri = f'{host_raw}:{port}'
+                    uri = f'https://{host_raw}:{port}'
                     connections.connect(alias=alias, uri=uri, token=token, timeout=5)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

                if token:
                    # Cloud: pass UI host and port verbatim via URI
                    uri = f'https://{host_raw}:{port}'
                    connections.connect(alias=alias, uri=uri, token=token, timeout=5)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_milvus/IGlobal.py` around lines 74 - 77, Update the
cloud validation branch in IGlobal.py to construct the URI with the same
https:// scheme used by the runtime Milvus client before calling
connections.connect. Preserve the raw host and port values while ensuring
token-based cloud validation receives a valid http[s]://host[:port] URI.
nodes/src/nodes/store_weaviate/IGlobal.py (1)

108-119: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Validation's cloud/local classification can diverge from the actual connection mode used at runtime.

validateConfig infers is_local/is_cloud purely from the host string (localhost/127.*), but the actual Store.__init__ in weaviate.py decides local vs. cloud from config.get('mode') (profile == 'cloud' else local). A mode=local deployment behind a non-loopback hostname (e.g. a Docker service name or private DNS entry) will be misclassified as cloud by this heuristic and probed via an HTTPS /v1/meta request with Bearer auth — which doesn't match what the node will actually do at runtime — producing a confusing/incorrect validation warning for a legitimately-configured local instance.

Prefer deriving is_cloud from config.get('mode') == 'cloud' (mirroring weaviate.py) instead of re-inferring it from the hostname.

🧰 Tools
🪛 ast-grep (0.44.1)

[warning] 113-113: Do not make http calls without encryption
Context: f'http://{host}'
Note: [CWE-319] Cleartext Transmission of Sensitive Information.

(requests-http)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_weaviate/IGlobal.py` around lines 108 - 119, Update
validateConfig’s cloud/local classification to derive is_cloud from
config.get('mode') == 'cloud', matching Store.__init__ in weaviate.py, and
derive is_local as its inverse. Remove the hostname-based localhost/127.*
heuristic while preserving the existing URL construction for each resulting
connection mode.
nodes/src/nodes/store_weaviate/README.md (1)

126-129: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Generated "View source" link still points to the pre-rename weaviate path.

Line 128 links to .../tree/develop/nodes/src/nodes/weaviate, but this node now lives at nodes/src/nodes/store_weaviate. Since this sits inside the generated params block, it shouldn't be hand-edited — as per path instructions, "do not hand-edit content between <!-- ROCKETRIDE:GENERATED:PARAMS START --> and <!-- ROCKETRIDE:GENERATED:PARAMS END --> as it is regenerated by nodes:docs-generate." Re-running the generator against the renamed folder should pick up the correct path.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_weaviate/README.md` around lines 126 - 129, Regenerate
the documentation for the renamed store_weaviate node using the
nodes:docs-generate workflow so the generated “View source” link targets
nodes/src/nodes/store_weaviate instead of the obsolete nodes/src/nodes/weaviate
path. Do not edit the content between the generated params markers manually.

Source: Path instructions

nodes/src/nodes/store_weaviate/requirements.txt (1)

1-1: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== requirements file =="
if [ -f nodes/src/nodes/store_weaviate/requirements.txt ]; then
  cat -n nodes/src/nodes/store_weaviate/requirements.txt
else
  fd -a 'requirements.txt' . | sed 's#^\./##' | head -50
fi

echo
echo "== authlib usages =="
rg -n "authlib|JWT|JWS|AuthCodeBackend|TokenAuth" nodes pyproject.toml requirements.txt 2>/dev/null || true

echo
echo "== pip index metadata if available =="
python3 - <<'PY'
import urllib.request, json, re
try:
    req = urllib.request.Request("https://pypi.org/pypi/authlib/json", headers={"User-Agent":"CodeRabbit verification"})
    data = json.load(urllib.request.urlopen(req, timeout=20))
    print("latest:", data["info"]["version"])
    versions = sorted(data["releases"], key=lambda v: tuple(map(int_or_inf, re.split(r'[.\-_]', v))))
    print("last 10 versions:", versions[-10:])
    for ver in versions[-30:]:
        print(ver, "yanked:", bool(data["releases"][ver][0].get("yanked")) if data["releases"][ver] else None)
except Exception as e:
    print("metadata fetch failed:", e)

def int_or_inf(s):
    try:
        return int(s)
    except:
        return float('inf')
PY

echo
echo "== pip index hashes for 1.3.1 and 1.7.2 =="
python3 - <<'PY'
import urllib.request, json, re
data = json.load(urllib.request.urlopen("https://pypi.org/pypi/authlib/json", timeout=20))
for ver in ("1.3.1", "1.7.2"):
    if ver in data.get("releases"):
        print(f"\n{ver}:")
        for rel in data["releases"][ver]:
            print(rel["packagetype"], rel["python_version"], rel["urls"] and rel["urls"][:1] if rel["urls"] else [], rel.get("yanked"))
        releases = data["releases"][ver]
        deps = []
        seen=set()
        for src in releases:
            deps.append(src.get("requires_dist",""))
        print("requires_dist:", list(set(deps)))
    else:
        print(f"{ver}: missing")
PY

Repository: rocketride-org/rocketride-server

Length of output: 50389


Pin authlib to 1.7.2 to avoid unresolved CVE-prone paths.

The leaf requirement still allows older resolved versions, while nodes/src/nodes/constraints.lock already pins authlib==1.7.2, so update this file to use the same patched minimum.

🛡️ Proposed fix
-authlib
+authlib>=1.7.2
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

authlib>=1.7.2
🧰 Tools
🪛 OSV Scanner (2.4.0)

[CRITICAL] 1-1: authlib 1.3.1: Authlib: JWS/JWT accepts unknown crit headers (RFC violation → possible authz bypass)

(PYSEC-2026-1200)


[CRITICAL] 1-1: authlib 1.3.1: Authlib has 1-click Account Takeover vulnerability

(PYSEC-2026-1201)


[CRITICAL] 1-1: authlib 1.3.1: Authlib : JWE zip=DEF decompression bomb enables DoS

(PYSEC-2026-1202)


[CRITICAL] 1-1: authlib 1.3.1: Authlib is vulnerable to Denial of Service via Oversized JOSE Segments

(PYSEC-2026-1203)


[CRITICAL] 1-1: authlib 1.3.1: undefined

(PYSEC-2026-188)


[CRITICAL] 1-1: authlib 1.3.1: undefined

(PYSEC-2026-2116)


[CRITICAL] 1-1: authlib 1.3.1: undefined

(PYSEC-2026-2117)


[CRITICAL] 1-1: authlib 1.3.1: undefined

(PYSEC-2026-2119)


[CRITICAL] 1-1: authlib 1.3.1: undefined

(PYSEC-2026-25)


[CRITICAL] 1-1: authlib 1.3.1: Authlib JWS JWK Header Injection: Signature Verification Bypass

(PYSEC-2026-287)


[CRITICAL] 1-1: authlib 1.3.1: Authlib Vulnerable to JWE RSA1_5 Bleichenbacher Padding Oracle

(GHSA-7432-952r-cw78)


[CRITICAL] 1-1: authlib 1.3.1: Authlib: JWS/JWT accepts unknown crit headers (RFC violation → possible authz bypass)

(GHSA-9ggr-2464-2j32)


[CRITICAL] 1-1: authlib 1.3.1: Authlib has 1-click Account Takeover vulnerability

(GHSA-fg6f-75jq-6523)


[CRITICAL] 1-1: authlib 1.3.1: Authlib : JWE zip=DEF decompression bomb enables DoS

(GHSA-g7f3-828f-7h7m)


[CRITICAL] 1-1: authlib 1.3.1: Authlib: Cross-site request forging when using cache

(GHSA-jj8c-mmj3-mmgv)


[CRITICAL] 1-1: authlib 1.3.1: Authlib: Fail-Open Cryptographic Verification in OIDC Hash Binding

(GHSA-m344-f55w-2m6j)


[CRITICAL] 1-1: authlib 1.3.1: Authlib is vulnerable to Denial of Service via Oversized JOSE Segments

(GHSA-pq5p-34cr-23v9)


[CRITICAL] 1-1: authlib 1.3.1: Authlib OIDC Implicit/Hybrid Authorization Vulnerable to Open Redirect

(GHSA-r95x-qfjj-fjj2)


[CRITICAL] 1-1: authlib 1.3.1: Authlib OAuth 2.0 has Open Redirect in Authorization API that allows attacker-controlled redirect_uri through unsupported response_type

(GHSA-w8p2-r796-3vmq)


[CRITICAL] 1-1: authlib 1.3.1: Authlib JWS JWK Header Injection: Signature Verification Bypass

(GHSA-wvwj-cvrp-7pv5)


[CRITICAL] 1-1: h11 0.8.1: h11 accepts some malformed Chunked-Encoding bodies

(PYSEC-2026-348)


[CRITICAL] 1-1: h11 0.8.1: h11 accepts some malformed Chunked-Encoding bodies

(GHSA-vqfr-h8mv-ghfj)


[MEDIUM] 1-1: h2 3.2.0: h2 allows HTTP Request Smuggling due to illegal characters in headers

(PYSEC-2026-1435)


[MEDIUM] 1-1: h2 3.2.0: h2 allows HTTP Request Smuggling due to illegal characters in headers

(GHSA-847f-9342-265h)


[HIGH] 1-1: idna 2.9.0: undefined

(PYSEC-2024-60)


[HIGH] 1-1: idna 2.9.0: undefined

(PYSEC-2026-215)


[HIGH] 1-1: idna 2.9.0: Internationalized Domain Names in Applications (IDNA): Specially crafted inputs to idna.encode() can bypass CVE-2024-3651 fix

(GHSA-65pc-fj4g-8rjx)


[HIGH] 1-1: idna 2.9.0: Internationalized Domain Names in Applications (IDNA) vulnerable to denial of service from specially crafted inputs to idna.encode

(GHSA-jjg7-2v4v-x38h)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_weaviate/requirements.txt` at line 1, Update the
authlib dependency declaration in requirements.txt to pin it to version 1.7.2,
matching the existing authlib==1.7.2 constraint in constraints.lock.
nodes/src/nodes/store_weaviate/weaviate.py (5)

267-277: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file excerpt =="
sed -n '230,295p' nodes/src/nodes/store_weaviate/weaviate.py | cat -n

echo "== search similarity/default/documentstore threshold =="
rg -n "similarity|threshold_search|distance|certainty|cosine" nodes src -S || true

echo "== python files outline around store weaviate =="
ast-grep outline nodes/src/nodes/store_weaviate/weaviate.py --view expanded || true

echo "== git status/stat =="
git diff --stat || true
git diff -- nodes/src/nodes/store_weaviate/weaviate.py | sed -n '1,220p' || true

echo "== behavioral probe for formulas =="
python3 - <<'PY'
import math, np
vals = [0, 0.5, 1, 1.5, 2]
for d in vals:
    current = (d + 1) / 2
    proposed = 1 - (d / 2)
    sigmoid = float(1.0 / (1.0 + math.exp(d / -100)))
    print(f"distance={d:.2f} current={current:.2f} proposed={proposed:.2f} sigmoid={sigmoid:.6f}")
PY

Repository: rocketride-org/rocketride-server

Length of output: 47150


🌐 Web query:

Weaviate cosine distance range 0 to 2 certainty formula 1 - distance / 2

💡 Result:

In Weaviate, when the cosine distance metric is used, the distance value ranges from 0 to 2 [1][2]. The certainty score is a normalized value between 0 and 1 that represents similarity, where 1 indicates identical vectors and 0 indicates opposite vectors [3][2]. The formula to convert a cosine distance to a certainty score is: certainty = 1 - (distance / 2) [4] This conversion is specific to the cosine distance metric [3][2]. Because not all distance metrics in Weaviate have a fixed range, the use of distance is generally preferred over certainty in modern Weaviate versions, though certainty remains supported for backward compatibility when using cosine distance [4][2].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
import math

def current(d): return (d + 1) / 2
def proposed(d): return 1 - (d / 2)
def sigmoid(d): return 1.0 / (1.0 + math.exp(d / -100))

for d in [0, 0.5, 1.0, 1.99, 2.0, 4.0]:
    print(f"distance={d:.2f} current={current(d):.2f} proposed={proposed(d):.2f} sigmoid={sigmoid(d):.6f}")
PY

echo "== deterministic property check =="
python3 - <<'PY'
def current(d): return (d + 1) / 2
def proposed(d): return 1 - (d / 2)
distances = [0, 1e-6, 0.20, 0.5, 1.0, 1.80, 1.999, 2.0]
for d in distances:
    print({
        'distance': d,
        'current_identity': current(d),
        'proposed_identity': proposed(d),
        'current_in_0_1': 0 <= current(d) <= 1,
        'proposed_in_0_1': 0 <= proposed(d) <= 1,
        'current_higher_for_d_greater_1': current(d) >= 1 for d in distances
    })
PY

Repository: rocketride-org/rocketride-server

Length of output: 737


Normalize Weaviate cosine distances with certainty semantics.

Weaviate cosine distances range from 0 to 2, so the current (distance + 1) / 2 mapping makes identical vectors score 0.5, passes dissimilar vectors like distance 1.8 as 1.4, and produces scores outside the [0, 1] threshold space. Use Weaviate’s cosine certainty formula: 1 - distance / 2.

🐛 Proposed fix
             if point.metadata.distance is not None:
                 if self.similarity == 'cosine':
-                    score = (point.metadata.distance + 1) / 2
+                    score = 1 - (point.metadata.distance / 2)
                 else:
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

            if point.metadata.distance is not None:
                if self.similarity == 'cosine':
                    score = 1 - (point.metadata.distance / 2)
                else:
                    score = float(1.0 / (1.0 + np.exp(point.metadata.distance / -100)))

                # Ignore it if it doesn't have a high enough score
                if score < 0.20:
                    continue
            else:
                score = 0.0
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_weaviate/weaviate.py` around lines 267 - 277, Update
the cosine branch in the score calculation around self.similarity to normalize
Weaviate distances using certainty semantics: compute the score as 1 minus
distance divided by 2. Keep the existing non-cosine calculation, threshold
check, and missing-distance behavior unchanged.

302-326: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Confirm QuestionText's __str__ behavior and how sibling stores build keyword filters from it.
rg -n -A5 -B2 'class QuestionText' --type=py
rg -n -A3 -B3 "searchKeyword" packages/ai/src nodes/src/nodes --type=py -g '!store_weaviate'

Repository: rocketride-org/rocketride-server

Length of output: 170


🏁 Script executed:

#!/bin/bash
set -e

echo "== files =="
git ls-files | rg '(^|/)weaviate\.py$|QuestionText|IGlobal|IGlobal|keyword'

echo "== target file outline/size =="
find . -path '*/store_weaviate/weaviate.py' -type f -print | while read f; do
  echo "FILE: $f"
  wc -l "$f"
  ast-grep outline "$f" --match searchKeyword --view expanded || true
done

echo "== QuestionText definitions/usages =="
rg -n -A10 -B5 'QuestionText' . --glob '*.py' || true

echo "== searchKeyword usages =="
rg -n -A8 -B6 'searchKeyword' . --glob '*.py' || true

echo "== keyword filter patterns in Python files =="
rg -n -A4 -B4 "like\\(f'\\*\\{|like\\(\\*|by_property\\('content'\\)" . --glob '*.py' || true

echo "== relevant target lines =="
for f in $(git ls-files | rg '(^|/)weaviate\.py$'); do
  echo "FILE: $f"
  sed -n '280,340p' "$f"
done

Repository: rocketride-org/rocketride-server

Length of output: 50388


🏁 Script executed:

#!/bin/bash
set -e

echo "== QuestionText definition =="
sed -n '262,295p' packages/client-python/src/rocketride/schema/question.py

echo "== store_weaviate relevant lines =="
sed -n '288,330p' nodes/src/nodes/store_weaviate/weaviate.py

echo "== sibling keyword filter implementations concise =="
python3 - <<'PY'
from pathlib import Path
paths = [
    'nodes/src/nodes/store_atlas/atlas.py',
    'nodes/src/nodes/store_chroma/chroma.py',
    'nodes/src/nodes/store_elasticsearch/elasticsearch_store.py',
    'nodes/src/nodes/store_milvus/milvus.py',
    'nodes/src/nodes/store_pinecone/pinecone.py',
    'nodes/src/nodes/store_postgres/postgres.py',
    'nodes/src/nodes/store_weaviate/weaviate.py',
]
text = "class DocumentStoreBase\n@abstractmethod\n    def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:"
for p in paths:
    s = Path(p).read_text()
    idx = s.find(text)
    if idx == -1:
        print(f"{p}: NOT FOUND")
        continue
    body = s[idx:s.find('\n    def searchSemantic', idx)]
    query_refs = sorted(set(l.strip() for l in body.splitlines() if 'query' in l or query := getattr(l,l, None)))
    print(f"\n--- {p} ---")
    print(body.strip())
PY

echo "== pydantic BaseModel repr/str probe if installed =="
python3 - <<'PY'
try:
    from pydantic import BaseModel
except Exception as e:
    print(f"pydantic import failed: {type(e).__name__}: {e}")
    raise SystemExit(0)
class QuestionText(BaseModel):
    text: str
    embedding_model: str = 'default'
    embedding: list | None = None
instance = QuestionText(text='hello')
print("Instance:", instance)
print("str:", str(instance))
print("repr:", repr(instance))
PY

Repository: rocketride-org/rocketride-server

Length of output: 3071


🏁 Script executed:

#!/bin/bash
set -e

echo "== concise sibling keyword implementations =="
python3 - <<'PY'
from pathlib import Path
marker = "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:"
markers = [
    ("nodes/src/nodes/store_atlas/atlas.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n        \"\"\"Perform keyword search using MongoDB text search.\"\"\""),
    ("nodes/src/nodes/store_chroma/chroma.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n        \"\"\"\n        Keyword search.\n        \"\"\""),
    ("nodes/src/nodes/store_elasticsearch/elasticsearch_store.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n        \"\"\"\n        Keyword search using Elasticsearch text search.\n        \"\"\""),
    ("nodes/src/nodes/store_milvus/milvus.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n        \"\"\"\n        Keyword search.\n        \"\"\""),
    ("nodes/src/nodes/store_pinecone/pinecone.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n        \"\"\"\n        Keyword search.\n        \"\"\""),
    ("nodes/src/nodes/store_postgres/postgres.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n        \"\"\"\n        Perform a keyword search."),
    ("nodes/src/nodes/store_weaviate/weaviate.py", "def searchKeyword(self, query: QuestionText, docFilter: DocFilter) -> List[Doc]:\n        \"\"\"\n        Keyword search.\n        \"\"\""),
]
for path, start in markers:
    s = Path(path).read_text()
    idx = s.find(start)
    if idx == -1:
        print(f"{path}: NOT FOUND")
        continue
    end = s.find("\n    def searchSemantic", idx)
    print(f"\n--- {path} ---")
    print(s[idx:end].strip())
PY

echo "== pydantic BaseModel repr/str probe if installed =="
python3 - <<'PY'
try:
    from pydantic import BaseModel
except Exception as e:
    print(f"pydantic import failed: {type(e).__name__}: {e}")
    raise SystemExit(0)
class QuestionText(BaseModel):
    text: str
instance = QuestionText(text='hello world')
print("Instance:", instance)
print("str:", str(instance))
print("repr:", repr(instance))
PY

echo "== pydantic install-only source facts if already installed =="
python3 - <<'PY'
import inspect
try:
    from pydantic import BaseModel
except Exception as e:
    print("not installed/unavailable")
    raise SystemExit(0)

cls = BaseModel
print("BaseModel MRO:", [c.__name__ for c in cls.mro()])
print("has __repr__:", hasattr(cls, '__repr__'))
try:
    print("__repr__ from:", inspect.getfile(cls.__repr__))
except Exception as e:
    print("__repr__ source unavailable:", e)
PY

Repository: rocketride-org/rocketride-server

Length of output: 13079


🌐 Web query:

Pydantic v2 BaseModel __str__ __repr__ default behavior ModelStr

💡 Result:

In Pydantic v2, BaseModel instances provide default implementations for str and repr to provide a human-readable representation of the model instance and its fields [1]. Default Behavior: - The repr method returns a string representation of the model, typically in the format ModelName(field1=value1, field2=value2,...) [1]. - The str method by default delegates to the repr method, meaning printing an instance of a Pydantic model will display the same information as its representation [2]. Customization: - Field Representation: You can control which fields appear in the output of str and repr using the Field(repr=...) argument [3]. By setting repr=False in a Field definition, that field will be excluded from the generated string/representation [3]. - Overriding: You can override str or repr directly in your BaseModel subclass if you need custom logic for how your model is displayed or printed [2]. Regarding ModelStr: - Pydantic does not have a native type called ModelStr. It is likely you are referring to the standard str type used as a type hint for fields, or potentially a misunderstanding of how string fields are handled [4][5]. Pydantic v2 handles standard Python strings as-is, while providing specialized string types like EmailStr or ImportString for specific validation needs [5]. If you were referring to an issue where custom subclassed string types appear to lose their specific class type information after model construction, this is a known aspect of Pydantic's design regarding validation and coercion [6].

Citations:


Use query.text in the Weaviate keyword filter.

QuestionText is a Pydantic model, so f'*{query}*' produces a model repr such as QuestionText(text=..., embedding_model=...) rather than the search text. Change the filter to Filter.by_property('content').like(f'*{query.text}*') to match the question text directly.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_weaviate/weaviate.py` around lines 302 - 326, The
searchKeyword method currently interpolates the entire QuestionText model into
the Weaviate content filter. Update the filter construction to use query.text so
the wildcard pattern matches the question text directly.

415-469: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Delete-then-insert in addChunks is not atomic; a partial batch failure can lose data.

Existing chunks for the incoming objectIds are deleted (line 442) before the new batch is inserted (445-463). If the batch import partially fails, the exception raised at line 467 surfaces only after the old chunks are already gone and some new chunks may be missing — leaving the document's chunk set corrupted with no rollback. The existing # TODO: Start discussion about better use of upsert() (line 436) hints at this being a known gap, though framed around performance rather than atomicity/data-loss risk.

Happy to help prototype a safer approach (e.g., insert-then-delete-old, or use Weaviate's UUID-based upsert so re-ingest overwrites in place without a delete step) if useful — want me to open an issue to track it?

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_weaviate/weaviate.py` around lines 415 - 469, Update
addChunks to avoid deleting existing chunks before the replacement batch has
been imported successfully. Use an atomic or rollback-safe replacement
strategy—prefer UUID-based upsert when available, otherwise insert the new
chunks first and delete prior chunks only after the batch completes without
failed_objects—while preserving the existing objectId replacement semantics.

489-514: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win

markDeleted/markActive silently truncate to 10 chunks — missing limit on fetch_objects.

Both methods call self.collectionObj.query.fetch_objects(filters=Filter.any_of(objectIdsFilter)) without a limit. The Weaviate v4 Python client defaults fetch_objects() to 10 results when no limit is given — confirmed on the community forum: "I have a simple collection with 1700 items... print(len(all_items)) -> this prints 10" unless limit is set. For any document with more than 10 chunks, only the first 10 will have isDeleted toggled, leaving the rest in the wrong soft-delete state — a silent, hard-to-detect data-integrity bug. Compare with render() in this same file, which already paginates in windows for exactly this reason.

🐛 Proposed fix: paginate until exhausted
-        objectIdsFilter: List[Filter] = []
-        for objectId in objectIds:
-            objectIdsFilter.append(Filter.by_property('objectId').equal(objectId))
-        points = self.collectionObj.query.fetch_objects(filters=Filter.any_of(objectIdsFilter))
-        for point in points.objects:
-            self.collectionObj.data.update(
-                uuid=point.uuid,
-                properties={
-                    'isDeleted': True,
-                },
-            )
-        return
+        objectIdsFilter: List[Filter] = []
+        for objectId in objectIds:
+            objectIdsFilter.append(Filter.by_property('objectId').equal(objectId))
+        combined_filter = Filter.any_of(objectIdsFilter)
+        while True:
+            points = self.collectionObj.query.fetch_objects(filters=combined_filter, limit=10000)
+            if not points.objects:
+                break
+            for point in points.objects:
+                self.collectionObj.data.update(uuid=point.uuid, properties={'isDeleted': True})
+            if len(points.objects) < 10000:
+                break
+        return

(apply the analogous change to markActive)

Also applies to: 516-541

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_weaviate/weaviate.py` around lines 489 - 514, Update
markDeleted and markActive so their fetch_objects queries do not rely on the
default 10-result limit. Paginate through matching objects in windows until no
results remain, applying the existing isDeleted update to every returned object.
Preserve the collection-existence and empty-objectIds early returns.

543-606: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win

render() preallocates a multi-million-element list on every window iteration.

text = [''] * self.renderChunkSize at line 573 allocates a Python list sized by renderChunkSize, whose default is 32 * 1024 * 1024 (~33.5M entries). Every call to render() — even for a document with a handful of chunks — allocates this ~33M-element list at least once (and repeats it per pagination window), burning significant CPU/memory on every render request. The naming (renderChunkSize) and magnitude (32*1024*1024, looking like a byte-size constant) suggest the "window" was never intended to hold literally that many chunks.

Consider keying by chunk - offset into a dict instead of preallocating a fixed-size list, so memory scales with the actual number of returned points rather than the configured window size.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/store_weaviate/weaviate.py` around lines 543 - 606, Update
render() to replace the fixed-size text list allocation with a sparse dictionary
keyed by each point’s chunk - offset, so memory usage scales with returned
chunks rather than renderChunkSize. Preserve chunk ordering, missing-chunk
handling, lastIndex/numberOfItems calculation, joined output, pagination, and
callback behavior.

@asclearuc asclearuc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dsapandora for the careful staging of this rename — the compat story holds up. I verified the core claim: node identity for saved pipes is the per-services.json protocol/title (proof: store_elasticsearch registers two providers from one folder), protocols/prefixes are untouched, and examples/db_arango.pipe still resolves. Redirects are complete and CI is fully green.

Requesting changes on the documentation half of the rename, per the co-located-docs rule (same change, not follow-up):

  1. 9 node READMEs keep the old identity — every renamed node except store_postgres still has the old name as its title (# milvus, # astra_db, # db_arango, …) and a "View source" link to the old folder path, which 404s once this merges. Please apply the same fix you did for store_postgres.
  2. docs/README-nodes.md catalog (lines 9, 137–145) still lists all nine old node IDs. On the Copilot thread you wrote it regenerates from services.json — I checked the tree and found no generator; docs.yml only runs builder docs:build, which doesn't touch this file. Copilot's comment stands; the table needs a manual update.
  3. tools/dependabot-smoke/smoke-nodes.sh:31 still keys nodes/src/nodes/index_search — that directory won't exist after merge, so the opensearchpy smoke check silently dies. Should be store_elasticsearch.
  4. PR description: still says vectordb_ prefix (actual: store_), says test folders follow the new names (they intentionally don't, per your own reply on the Copilot thread), and counts eight stores (there are nine). Please refresh it — it's what release notes will show.

Non-blocking: a few comments made stale by the rename (graph_falkordb/services.json:203, graph_hydradb/services.json:16-17, packages/ai/.../graph/__init__.py:30) and content-static links to old /nodes/* URLs (redirects cover them).

Co-located-docs follow-through for the rename in this PR (per asclearuc review):
- 10 node READMEs still carried the old title and a "View source" link to the
  old folder (# milvus -> # store_milvus, .../nodes/milvus ->
  .../nodes/store_milvus), which 404s after merge. store_postgres was already done.
- docs/README-nodes.md: the Vector Databases & Stores table listed the old
  folder-derived ids; refreshed to store_*. The catalog is hand-maintained (no
  generator), so the misleading "generated" note was corrected to match.
- tools/dependabot-smoke/smoke-nodes.sh: index_search -> store_elasticsearch so
  the opensearchpy smoke check keeps resolving after the folder rename.
Now that #1642's collection-time guard undoes the sys.modules stub leaks, the
reordered pytest collection is harmless, so the test folders can follow their
nodes: nodes/test/{astra_db,atlas,chroma,milvus,pinecone,qdrant,weaviate} ->
store_*. Tests load src via ../../src/nodes/store_* (unchanged depth), so only
the two ocr breadcrumbs and two stale "Usage" docstrings needed the new paths.
Copilot AI review requested due to automatic review settings July 23, 2026 11:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 43 out of 128 changed files in this pull request and generated no new comments.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)
nodes/src/nodes/graph_hydradb/README.md (2)

66-67: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Regenerate the missing configuration schema.

The generated-parameters block is empty, so published documentation omits the node’s API key, database, collection, and max-results schema. Run nodes:docs-generate and commit the generated content; do not hand-edit between these markers.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/graph_hydradb/README.md` around lines 66 - 67, Regenerate the
missing configuration schema in the ROCKETRIDE:GENERATED:PARAMS block by running
nodes:docs-generate, then commit the generated API key, database, collection,
and max-results documentation without hand-editing content between the markers.

Source: Coding guidelines


27-27: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

Do not promise memory zeroization without implementing it.

endGlobal() only assigns self.client.api_key = '' and drops the client reference, which does not guarantee all existing secret copies are removed. Either implement actual secret zeroization or change the README/IGlobal.py wording to say the client/key reference is cleared/dropped.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/src/nodes/graph_hydradb/README.md` at line 27, Update the API-key
lifecycle wording in README.md and the corresponding IGlobal.py documentation to
state only that the client/key reference is cleared or dropped during teardown,
unless endGlobal() is changed to implement actual zeroization of every secret
copy. Remove the unsupported promise that the key is cleared from memory.
nodes/test/store_milvus/test_filter_injection.py (1)

93-97: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

Assert escaping rather than remove the payload under test.

Line 97 removes '|| 1==1' before checking for it, so the assertion always passes regardless of sanitization.

Proposed fix
-        self.assertNotIn('|| 1==1', filter_expr.replace(_escape_milvus_str('|| 1==1'), ''))
+        self.assertEqual(filter_expr, "content like '%%\\' || 1==1 || content like \\'%%'")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@nodes/test/store_milvus/test_filter_injection.py` around lines 93 - 97,
Update test_inject_via_keyword_search so it asserts the constructed filter_expr
directly, without replacing or removing the payload fragment under test. Verify
that the escaped result does not contain the unescaped injection sequence while
preserving the existing payload and _escape_milvus_str setup.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@nodes/test/store_milvus/test_addchunks_return.py`:
- Line 19: Enable the regression test by removing the pytest skip marker and
refactoring its imports or fixtures as needed so it runs successfully. Preserve
the test’s coverage of the addChunks() return contract.

In `@nodes/test/store_milvus/test_filter_injection.py`:
- Line 14: Scope the temporary sys.path insertion around the milvus provider
import and restore the original import path immediately afterward in
nodes/test/store_milvus/test_filter_injection.py lines 14-14. Apply the same
scoped cleanup around the provider import in
nodes/test/store_milvus/test_search_semantic.py lines 13-13, ensuring test
collection does not retain either path modification.

---

Outside diff comments:
In `@nodes/src/nodes/graph_hydradb/README.md`:
- Around line 66-67: Regenerate the missing configuration schema in the
ROCKETRIDE:GENERATED:PARAMS block by running nodes:docs-generate, then commit
the generated API key, database, collection, and max-results documentation
without hand-editing content between the markers.
- Line 27: Update the API-key lifecycle wording in README.md and the
corresponding IGlobal.py documentation to state only that the client/key
reference is cleared or dropped during teardown, unless endGlobal() is changed
to implement actual zeroization of every secret copy. Remove the unsupported
promise that the key is cleared from memory.

In `@nodes/test/store_milvus/test_filter_injection.py`:
- Around line 93-97: Update test_inject_via_keyword_search so it asserts the
constructed filter_expr directly, without replacing or removing the payload
fragment under test. Verify that the escaped result does not contain the
unescaped injection sequence while preserving the existing payload and
_escape_milvus_str setup.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 71357b06-1e9b-4ef8-a2a9-33055b5a488b

📥 Commits

Reviewing files that changed from the base of the PR and between 132437a and e6bf96e.

📒 Files selected for processing (31)
  • docs/README-nodes.md
  • nodes/src/nodes/graph_arango/README.md
  • nodes/src/nodes/graph_hydradb/README.md
  • nodes/src/nodes/store_astra/README.md
  • nodes/src/nodes/store_atlas/README.md
  • nodes/src/nodes/store_chroma/README.md
  • nodes/src/nodes/store_elasticsearch/README.md
  • nodes/src/nodes/store_milvus/README.md
  • nodes/src/nodes/store_pinecone/README.md
  • nodes/src/nodes/store_qdrant/README.md
  • nodes/src/nodes/store_weaviate/README.md
  • nodes/test/ocr/test_model_server_ocr.py
  • nodes/test/store_astra/__init__.py
  • nodes/test/store_astra/test_convert_filter.py
  • nodes/test/store_atlas/__init__.py
  • nodes/test/store_atlas/test_isdeleted_filter.py
  • nodes/test/store_chroma/__init__.py
  • nodes/test/store_chroma/test_coerce_port.py
  • nodes/test/store_chroma/test_convert_filter_explicit_none.py
  • nodes/test/store_milvus/__init__.py
  • nodes/test/store_milvus/test_addchunks_return.py
  • nodes/test/store_milvus/test_filter_injection.py
  • nodes/test/store_milvus/test_search_semantic.py
  • nodes/test/store_pinecone/__init__.py
  • nodes/test/store_pinecone/test_record_updates.py
  • nodes/test/store_qdrant/__init__.py
  • nodes/test/store_qdrant/test_mark_deleted.py
  • nodes/test/store_qdrant/test_search_threshold.py
  • nodes/test/store_weaviate/__init__.py
  • nodes/test/store_weaviate/test_record_updates.py
  • tools/dependabot-smoke/smoke-nodes.sh

@dsapandora

Copy link
Copy Markdown
Collaborator Author

Thanks Alexandru — all fixed on e6bf96eb (all green).

READMEs (titles + View-source links), the README-nodes.md catalog, and smoke-nodes.sh are all pointing at the new names now (bce1ba8a). And you're right that I was wrong on the Copilot thread — there's no generator, gather.js only collects the READMEs, so I updated the catalog by hand and fixed that "generated" note.

Also went one further: now that #1642's guard kills the sys.modules leaks, I renamed the 7 test folders to store_* too (e6bf96eb) — nothing keeps the old identity anymore. Description refreshed as well.

Left the non-blocking stale comments out to keep this renames-only; happy to fold them in if you'd rather.

@asclearuc asclearuc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @dsapandora
Approving

@dsapandora
dsapandora enabled auto-merge (squash) July 23, 2026 13:17
@dsapandora
dsapandora merged commit ddf94dd into develop Jul 23, 2026
30 checks passed
@dsapandora
dsapandora deleted the refactor/node-family-renames branch July 23, 2026 13:18
charliegillet added a commit that referenced this pull request Aug 1, 2026
…s/ variant dir (#1757)

* fix(docs): unblock docs deploys broken since #1570

docs:gather's package-mount pass globs {nodes,packages,apps}/**/docs/** and
throws on any match without a declared mount. tool_google_workspace's Google
Docs variant lives at nodes/src/nodes/tool_google_workspace/docs/, so its
README matches the glob and every docs build since 2026-07-17 has aborted,
leaving docs.rocketride.org stale since the last successful deploy on 07-15.

Exclude nodes/src/nodes/** and nodes/test/** from the package-mount pass;
markdown under node sources is staged by the dedicated nodes contributor.

Also fix the stale internal links that the revived build surfaces: four spine
pages still linked pre-rename routes from #1636/#1611 (/nodes/qdrant,
/nodes/pinecone, /nodes/milvus, /nodes/db_neo4j), which now fail the
broken-links check. Point them at the store_*/graph_* routes and update the
neo4j page's node name.

Verified locally: builder docs:build compiles clean; gather stages 170 pages
including all 118 node docs.

Fixes #1756

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(build): correct the IGNORE comment about who stages node markdown

Review pointed out the comment overclaimed: variant READMEs are only staged
when the node's top-level README carries the generated markers, so
tool_google_workspace's variant docs are currently staged by nobody. Say
what the contributor pass actually does instead of implying full coverage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
AviNormie pushed a commit to AviNormie/rocketride-server that referenced this pull request Nov 9, 2026
…o graph_* (rocketride-org#1636)

* refactor(nodes): rename vector stores to store_* and graph-first db to graph_*

Rename the nine vector-store nodes to the store_ prefix (matching their `store`
classType) and the two graph-first databases (ArangoDB, HydraDB) to graph_.
Update each services.json path to the new python module and add graph to the
ArangoDB/HydraDB classType (database and tool kept). protocol and prefix are left
untouched, so saved pipes keep resolving; no services-version bump.

Docs redirects map the old /nodes/<old> URLs to the new ones. Test module imports
and path references follow the renamed src folders. constraints.lock and docs
static pages are regenerated by CI.

Node test folders keep their original names so pytest collection order is unchanged.

* docs(nodes): align renamed node docs with store_*/graph_* identity

Co-located-docs follow-through for the rename in this PR (per asclearuc review):
- 10 node READMEs still carried the old title and a "View source" link to the
  old folder (# milvus -> # store_milvus, .../nodes/milvus ->
  .../nodes/store_milvus), which 404s after merge. store_postgres was already done.
- docs/README-nodes.md: the Vector Databases & Stores table listed the old
  folder-derived ids; refreshed to store_*. The catalog is hand-maintained (no
  generator), so the misleading "generated" note was corrected to match.
- tools/dependabot-smoke/smoke-nodes.sh: index_search -> store_elasticsearch so
  the opensearchpy smoke check keeps resolving after the folder rename.

* test(nodes): rename test folders to store_* to match the renamed nodes

Now that rocketride-org#1642's collection-time guard undoes the sys.modules stub leaks, the
reordered pytest collection is harmless, so the test folders can follow their
nodes: nodes/test/{astra_db,atlas,chroma,milvus,pinecone,qdrant,weaviate} ->
store_*. Tests load src via ../../src/nodes/store_* (unchanged depth), so only
the two ocr breadcrumbs and two stale "Usage" docstrings needed the new paths.
AviNormie pushed a commit to AviNormie/rocketride-server that referenced this pull request Nov 9, 2026
…s/ variant dir (rocketride-org#1757)

* fix(docs): unblock docs deploys broken since rocketride-org#1570

docs:gather's package-mount pass globs {nodes,packages,apps}/**/docs/** and
throws on any match without a declared mount. tool_google_workspace's Google
Docs variant lives at nodes/src/nodes/tool_google_workspace/docs/, so its
README matches the glob and every docs build since 2026-07-17 has aborted,
leaving docs.rocketride.org stale since the last successful deploy on 07-15.

Exclude nodes/src/nodes/** and nodes/test/** from the package-mount pass;
markdown under node sources is staged by the dedicated nodes contributor.

Also fix the stale internal links that the revived build surfaces: four spine
pages still linked pre-rename routes from rocketride-org#1636/rocketride-org#1611 (/nodes/qdrant,
/nodes/pinecone, /nodes/milvus, /nodes/db_neo4j), which now fail the
broken-links check. Point them at the store_*/graph_* routes and update the
neo4j page's node name.

Verified locally: builder docs:build compiles clean; gather stages 170 pages
including all 118 node docs.

Fixes rocketride-org#1756

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(build): correct the IGNORE comment about who stages node markdown

Review pointed out the comment overclaimed: variant READMEs are only staged
when the node's top-level README carries the generated markers, so
tool_google_workspace's variant docs are currently staged by nobody. Say
what the contributor pass actually does instead of implying full coverage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation module:nodes Python pipeline nodes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants