Skip to content

fix(atlas): add missing isDeleted filter in _convertFilter and guard empty objectIds in mutations - #367

Merged
stepmikhaylov merged 2 commits into
rocketride-org:developfrom
krishhgg:bugfix/atlas-isdeleted-filter
Mar 26, 2026
Merged

stepmikhaylov merged 2 commits into
rocketride-org:developfrom
krishhgg:bugfix/atlas-isdeleted-filter

Conversation

@krishhgg

@krishhgg krishhgg commented Mar 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add isDeleted handling to Atlas MongoDB _convertFilter, aligning it with every other vector store (Qdrant, Weaviate, Milvus, Pinecone, Postgres, Chroma, Astra_DB, Elasticsearch)
  • Add empty-list guards to remove, markDeleted, markActive to short-circuit on objectIds=[]
  • Add 12 unit tests covering both fixes

Bug

The Atlas MongoDB vector store node (nodes/src/nodes/atlas/atlas.py) has two bugs:

  1. Missing isDeleted filter in _convertFilter — Every other vector store handles DocFilter.isDeleted in its _convertFilter method, but Atlas does not. This causes:

    • Soft-deleted documents leaking into search and get() results. Users who soft-delete documents via markDeleted() will still see those documents in keyword search, semantic search, and get() queries.
    • Broken _checkCollectionExists() in the base class — The base class DocumentStoreBase._checkCollectionExists() (line 421 of store.py) sets filter.isDeleted = True to locate the schema control document. Without isDeleted filtering in _convertFilter, the query get(filter) ignores the isDeleted=True flag entirely, returning all docs with objectId='schema' regardless of deletion status. While this may not immediately break (if only one such document exists), it violates the contract and can return incorrect results if stale control documents exist.
  2. Missing empty-list guards in remove, markDeleted, markActive — When called with an empty objectIds list, MongoDB's {'$in': []} matches nothing, which is safe but sends an unnecessary round-trip to the server. This is a minor defensive fix for consistency with other nodes.

Steps to Reproduce

from ai.common.schema import DocFilter

# 1. Create an Atlas store and add some documents
store = Store('atlas', connConfig, bag)
store.addChunks(chunks)

# 2. Soft-delete a document
store.markDeleted(['doc-123'])

# 3. Search — EXPECTED: doc-123 excluded. ACTUAL: doc-123 still appears
results = store.get(DocFilter())  # isDeleted defaults to None
# doc-123 is in results because _convertFilter never adds meta.isDeleted=False

# 4. Check collection existence — uses isDeleted=True internally
# filter.isDeleted = True is set but ignored by _convertFilter
exists = store.doesCollectionExist()

Filter output comparison (before vs after fix):

# BEFORE (broken): DocFilter() produces:
{}  # No isDeleted constraint — deleted docs leak through

# AFTER (fixed): DocFilter() produces:
{'meta.isDeleted': False}  # Deleted docs excluded

Root Cause

In _convertFilter (line 396–431 of atlas.py), the method converts every DocFilter field to a MongoDB query condition except isDeleted. The field is simply never checked.

Compare with the identical pattern implemented in:

  • Qdrant (qdrant.py line 215-216): if docFilter.isDeleted is None or not docFilter.isDeleted: must.append(FieldCondition(key='meta.isDeleted', match=MatchValue(value=False)))
  • Weaviate (weaviate.py line 226-227): if docFilter.isDeleted is None or not docFilter.isDeleted: must.append(Filter.by_property('isDeleted').equal(False))
  • Milvus (milvus.py line 208-209): if docFilter.isDeleted is None or not docFilter.isDeleted: must_conditions.append("meta['isDeleted'] == False")

Atlas is the only vector store missing this filter.

Fix

nodes/src/nodes/atlas/atlas.py:

  1. Added isDeleted handling to _convertFilter after the objectIds block:

    if docFilter.isDeleted is None or not docFilter.isDeleted:
        filter_conditions['meta.isDeleted'] = False
  2. Added empty-list early returns to remove, markDeleted, and markActive:

    if not objectIds:
        return

Why This Works

  • When isDeleted is None (default) or False: the filter adds {'meta.isDeleted': False}, excluding soft-deleted documents from results. This is the standard behavior across all other vector stores.
  • When isDeleted is True: no filter is added, so all documents (including deleted ones) are returned. This is exactly what the base class needs in _checkCollectionExists() to find the schema control document (which is stored with isDeleted=True).
  • The empty-list guards prevent unnecessary MongoDB round-trips and match the defensive pattern used in Chroma and Weaviate.

Testing / Validation

12 new unit tests in nodes/test/test_atlas_isdeleted_filter.py:

Test What it verifies
test_default_filter_excludes_deleted DocFilter() → {'meta.isDeleted': False}
test_explicit_false_excludes_deleted DocFilter(isDeleted=False) → {'meta.isDeleted': False}
test_explicit_true_includes_all DocFilter(isDeleted=True) → no meta.isDeleted key
test_combined_with_nodeId Both nodeId and isDeleted filters coexist
test_combined_with_objectIds Both objectIds and isDeleted filters coexist
test_isdeleted_true_with_objectIds isDeleted=True + objectIds → only objectIds filter
test_remove_noop_on_empty_list remove([]) → no MongoDB call
test_markDeleted_noop_on_empty_list markDeleted([]) → no MongoDB call
test_markActive_noop_on_empty_list markActive([]) → no MongoDB call
test_remove_calls_mongodb_with_ids remove(['id-1']) → correct delete_many call
test_markDeleted_calls_mongodb_with_ids markDeleted(['id-1']) → correct update_many call
test_markActive_calls_mongodb_with_ids markActive(['id-1']) → correct update_many call
# All tests pass (no MongoDB server required)
$ python3 -m pytest nodes/test/test_atlas_isdeleted_filter.py -v
12 passed in 0.03s

# Contract tests pass
$ python3 -m pytest nodes/test/test_contracts.py -v -k atlas
6 passed, 306 deselected in 0.05s

# Linter clean
$ ruff check nodes/src/nodes/atlas/atlas.py nodes/test/test_atlas_isdeleted_filter.py
All checks passed!

./builder test requires a full native C++ build chain (CMake, vcpkg, JDK 17) which is not feasible in this environment. The Python unit tests and contract tests cover the changed code paths.

Type

  • Bug fix

Checklist

  • Tests added or updated (nodes/test/test_atlas_isdeleted_filter.py — 12 tests)
  • Tested locally (pytest + ruff)
  • ./builder test passes (blocked by native C++ build requirement; Python tests and contract tests pass)
  • Commit messages follow conventional commits (fix(atlas): ...)
  • No secrets or credentials included

Related Issues

None filed — fix identified by parity review with other vector store implementations.

#frontier-tower-hackathon

Impact

This aligns Atlas filtering with other stores so soft-deleted documents do not leak into normal query results.

CI Evidence

Summary by CodeRabbit

  • Bug Fixes

    • Fixed deletion status filter behavior to ensure consistent and accurate query results.
  • Tests

    • Added comprehensive test coverage for filter construction and data mutation operations.

@github-actions github-actions Bot added the module:nodes Python pipeline nodes label Mar 22, 2026
@coderabbitai

coderabbitai Bot commented Mar 22, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Early-return guards were added to remove(), markDeleted(), and markActive() methods to prevent unnecessary database operations when objectIds is empty. The _convertFilter() method now consistently includes meta.isDeleted: False in filter conditions when isDeleted is not explicitly True, with comprehensive unit tests added to validate this behavior.

Changes

Cohort / File(s) Summary
Atlas Store Filter & Guard Logic
nodes/src/nodes/atlas/atlas.py
Added early-return guards to mutation methods (remove, markDeleted, markActive) to skip no-op database calls when objectIds is empty. Updated _convertFilter() to always inject meta.isDeleted: False constraint when docFilter.isDeleted is None or falsy.
Atlas Store Filter Tests
nodes/test/atlas/test_isdeleted_filter.py
New test module with mocked dependencies validating filter construction behavior (isDeleted flag handling, field combinations) and verifying empty-list short-circuiting for mutation methods alongside correct query/update payloads for non-empty inputs.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • jmaionchi
  • Rod-Christensen

Poem

🐰 A rabbit hops through filters clean,
Guards now catch what once was lean,
No empty queries dance in vain,
With isDeleted logic plain!
Thump thump—efficiency reigns! ✨

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes both main fixes: adding missing isDeleted filter in _convertFilter and guarding empty objectIds in mutations, matching the actual changeset.
Docstring Coverage ✅ Passed Docstring coverage is 94.44% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@krishhgg krishhgg changed the title fix(atlas): honor soft-delete filter and guard empty object id mutations fix(atlas): add missing isDeleted filter in _convertFilter and guard empty objectIds in mutations Mar 22, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@nodes/test/test_atlas_isdeleted_filter.py`:
- Around line 53-66: The __init__ in the filter class uses redundant
kwargs.get(..., None) calls; update the constructor to call kwargs.get('nodeId')
etc. (remove the explicit None defaults) for attributes nodeId, isTable,
tableIds, parent, permissions, objectIds, isDeleted, chunkIds, minChunkId,
maxChunkId while leaving the explicit defaults for limit and offset
(kwargs.get('limit', 25), kwargs.get('offset', 0)) unchanged; ensure assignments
like self.nodeId = kwargs.get('nodeId') and similarly for the other listed
attributes.
- Around line 147-148: The two nested context managers should be combined into a
single with statement to satisfy Ruff SIM117: replace the nested with
patch.object(atlas_mod.Config, 'getNodeConfig', return_value=mock_config) and
with patch.object(atlas_mod, 'MongoClient') as MockClient: with one combined
context manager using a comma (i.e., with patch.object(atlas_mod.Config,
'getNodeConfig', return_value=mock_config), patch.object(atlas_mod,
'MongoClient') as MockClient:) so the test uses a single with block while
preserving MockClient and the mocked getNodeConfig behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6b311701-1473-4e36-88b7-bb04b9ae4dcd

📥 Commits

Reviewing files that changed from the base of the PR and between 69894a2 and 426a10b.

📒 Files selected for processing (2)
  • nodes/src/nodes/atlas/atlas.py
  • nodes/test/test_atlas_isdeleted_filter.py

@krishhgg

Copy link
Copy Markdown
Collaborator Author

Evidence hardening pass for this PR.

Reproduction (before)

  • Query Atlas records using default/explicit isDeleted combinations and run mutation operations with an empty objectIds list.
  • Before the fix, _convertFilter could miss isDeleted semantics and mutation methods could proceed with empty identifiers.

Validation (after)

  • ../../.venv/bin/pytest nodes/test/test_atlas_isdeleted_filter.py -q
  • Outcome: 12 passed

CI Evidence

Why this is distinct

  • This PR combines Atlas filter correctness (isDeleted) with empty-objectIds mutation guarding in Atlas-specific code paths.

kwit75
kwit75 previously approved these changes Mar 24, 2026

@kwit75 kwit75 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Adds isDeleted filter + guards empty objectIds in atlas mutations.

@stepmikhaylov stepmikhaylov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kgarg2468

Thanks for this contribution.

One small request before merging: could you please move nodes/test/test_atlas_isdeleted_filter.py to nodes/test/atlas/test_isdeleted_filter.py so the test layout matches our project conventions?

- Add meta.isDeleted=false to Mongo filters when DocFilter does not request
  deleted docs, matching qdrant/weaviate/milvus behavior
- Short-circuit remove/markDeleted/markActive when objectIds is empty
- Add unit tests for _convertFilter and mutation guards

Made-with: Cursor
@stepmikhaylov
stepmikhaylov force-pushed the bugfix/atlas-isdeleted-filter branch from b269ad8 to ef959f4 Compare March 26, 2026 01:37
@stepmikhaylov
stepmikhaylov self-requested a review March 26, 2026 01:38

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@nodes/test/atlas/test_isdeleted_filter.py`:
- Around line 29-43: The test currently injects globals into sys.modules using
sys.modules.setdefault (via mock_depends_mod and multiple pymongo/numpy
entries), which can leak mocks across tests; replace these top-level setdefault
calls with a scoped patch by using unittest.mock.patch.dict on sys.modules
around the import/load of the atlas module (or inside a test fixture) so the
temporary module entries are applied only during atlas module loading and
automatically reverted; specifically, encapsulate the creation of
mock_depends_mod and the sys.modules entries for 'depends', 'pymongo',
'pymongo.collection', 'pymongo.database', 'pymongo.errors',
'pymongo.operations', and 'numpy' in a patch.dict context/fixture used when
importing the code that references those modules (e.g., where atlas is
imported/initialized) to avoid global pollution.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: c9a3cda8-2b17-4e2e-bd3a-c478ace95a15

📥 Commits

Reviewing files that changed from the base of the PR and between 426a10b and ef959f4.

📒 Files selected for processing (3)
  • nodes/src/nodes/atlas/atlas.py
  • nodes/test/atlas/__init__.py
  • nodes/test/atlas/test_isdeleted_filter.py

@stepmikhaylov
stepmikhaylov enabled auto-merge (squash) March 26, 2026 03:39
@stepmikhaylov
stepmikhaylov self-requested a review March 26, 2026 03:40
@stepmikhaylov
stepmikhaylov self-requested a review March 26, 2026 13:50
@stepmikhaylov
stepmikhaylov merged commit 1af45bc into rocketride-org:develop Mar 26, 2026
14 of 18 checks passed
AviNormie pushed a commit to AviNormie/rocketride-server that referenced this pull request Nov 9, 2026
…empty objectIds in mutations (rocketride-org#367)

* fix(atlas): honor soft-delete filter and guard empty object id mutations

- Add meta.isDeleted=false to Mongo filters when DocFilter does not request
  deleted docs, matching qdrant/weaviate/milvus behavior
- Short-circuit remove/markDeleted/markActive when objectIds is empty
- Add unit tests for _convertFilter and mutation guards

Made-with: Cursor

* refactor: move atlas test to nodes/test/atlas per project conventions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:nodes Python pipeline nodes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants