Skip to content

fix(nodes): send store_milvus metadata as a plain dict for pymilvus 3 - #3032

Closed
ptlnextdoor wants to merge 3 commits into
rocketride-org:developfrom
ptlnextdoor:fix/milvus-json-meta
Closed

ptlnextdoor wants to merge 3 commits into
rocketride-org:developfrom
ptlnextdoor:fix/milvus-json-meta

Conversation

@ptlnextdoor

@ptlnextdoor ptlnextdoor commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #3014

Root cause

Store.addChunks in nodes/src/nodes/store_milvus/milvus.py put chunk.metadata, a pydantic DocMetadata, straight into the JSON field meta. pymilvus 3 serializes JSON fields itself and rejects the model (DataNotMatchException: {meta} field should be a json), so every write failed, starting with the control row that createCollection writes. That left an empty collection, which is why later runs raised Collection does not have control document, found 0. The row now carries chunk.metadata.model_dump(), the same conversion store_postgres already uses. The read path is unchanged because it already handles meta as a dict.

Tests

The Milvus stub in nodes/test/test_store_object_batches.py now reads meta as a dict, the way a real server returns it. A new test, test_milvus_rows_carry_json_serializable_meta, checks that every upserted meta survives json.dumps.

python -m pytest nodes/test/test_store_object_batches.py nodes/test/store_milvus -q
44 passed, 1 skipped

Without the source change, the same command gives 5 failed (the new test plus the four Milvus cases). ruff check and ruff format --check pass on both files. I ran the tests locally with depends/engLib stubbed on PYTHONPATH (no built engine) and did not test against a live Milvus server.

Summary by CodeRabbit

  • Bug Fixes
    • Improved reliability when storing content in Milvus by ensuring chunk metadata remains compatible with JSON-based storage.
    • Preserved chunk identifiers in stored metadata, supporting consistent retrieval and processing.

Copilot AI balanced review requested due to automatic review settings November 5, 2026 03:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@github-actions github-actions Bot added module:nodes Python pipeline nodes tests labels Nov 5, 2026
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Milvus upsert rows now use dictionary-form chunk metadata instead of metadata model instances. The test stub reads metadata as dictionary keys, and a test checks that JSON serialization preserves chunk IDs.

Changes: Milvus metadata serialization

Layer / File(s) Summary
Serialize metadata for Milvus writes
nodes/src/nodes/store_milvus/milvus.py, nodes/test/test_store_object_batches.py
addChunks converts chunk metadata with model_dump() before adding it to an upsert row. The test stub reads metadata as dictionary keys, and a test checks JSON serialization preserves chunk IDs [0, 1].

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: High

Suggested labels: module:nodes, tests

Merge Risk: Minimal · up to b9ba2

Merge Risk: ⚪ Minimal · up to b9ba2

The change appears mergeable; live Milvus validation remains a normal follow-up rather than a confirmed merge-blocking risk.

Pre-merge checks: Passed 5

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: passing Milvus metadata as a plain dictionary for pymilvus 3.
Linked Issues check Passed Issue [#3014] requires JSON-compatible meta values for store_milvus writes. nodes/src/nodes/store_milvus/milvus.py now passes chunk.metadata.model_dump() to the Milvus meta field. The new te…
Out of Scope Changes check Passed The pull request changes only the Milvus metadata conversion and the related Milvus test stub and regression test. The test changes support issue [#3014] by modeling dictionary responses and validatin…
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Full details: Linked Issues check

Explanation

Issue [#3014] requires JSON-compatible meta values for store_milvus writes. nodes/src/nodes/store_milvus/milvus.py now passes chunk.metadata.model_dump() to the Milvus meta field. The new test in nodes/test/test_store_object_batches.py serializes every upserted meta with json.dumps and preserves chunkId values. This addresses the failed control-row and chunk writes. The reported lack of live Milvus testing does not show a coding requirement failure.

Full details: Out of Scope Changes check

Explanation

The pull request changes only the Milvus metadata conversion and the related Milvus test stub and regression test. The test changes support issue [#3014] by modeling dictionary responses and validating JSON serialization. No unrelated change is shown in the whole-pull-request diff.

Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 2 files.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between a7f3770 and b6351da.

Files selected for processing: 2

  • nodes/src/nodes/store_milvus/milvus.py
  • nodes/test/test_store_object_batches.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor
🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

@ptlnextdoor

Copy link
Copy Markdown
Contributor Author

CI note: the failing jobs are Builder failed: AggregateError [ETIMEDOUT] while downloading externals, before any test from this change runs. The nightly check-externals on develop failed the same way at 03:09 UTC today, so it looks like an upstream download outage. Happy to rebase once it clears. The node tests touched here pass locally (pytest nodes/test/test_store_object_batches.py).

@asclearuc

Copy link
Copy Markdown
Collaborator

@ptlnextdoor Thank you for this fix. #3055 is now merged and makes the same meta change, so I am closing this PR.

One detail for reference: model_dump() keeps the empty fields as None. When a row is read back, Doc rejects a None for fields such as nodeId and parent, so every read failed against a real Milvus. #3055 stores DocMetadata.toDict(), which leaves the empty fields out.

@asclearuc asclearuc closed this Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:nodes Python pipeline nodes tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

store_milvus: every write to a real Milvus fails with pymilvus 3 ({meta} field should be a json)

3 participants