Repository navigation
fix(nodes, test): scope core-module stubs so they don't leak across tests - #1642
Conversation
🤖 Internal: Discord sync markerAuto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughTest import helpers now scope mocked dependency modules to module loading and restore prior ChangesTest import isolation
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR improves pytest isolation for node/unit tests by scoping hand-rolled stubs for engine-only/core modules to the narrow import window that needs them, and restoring sys.modules immediately afterward. This prevents cross-test contamination when collection/execution order changes.
Changes:
- Wrapped core-module stubbing (
rocketlib,pydantic,ai.common.*,requests,tenacity) intry/finallyblocks that restore priorsys.modulesentries. - Replaced several
sys.modules.setdefault(...)usages with direct assignment to avoid reusing previously-contaminated stubs. - Applied the pattern across multiple test modules that import node code under synthetic module/package setups.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| nodes/test/tool_xtrace_memory/test_tools.py | Save/restore core stubs around node imports to prevent sys.modules leaks. |
| nodes/test/tool_mem0/test_tools.py | Save/restore core stubs around node imports to prevent sys.modules leaks. |
| nodes/test/tool_filesystem/test_read_size_cap.py | Scope rocketlib stub strictly to the node import and restore afterward. |
| nodes/test/test_currency_convert_explicit.py | Scope rocketlib stub to the node import and restore afterward; switch to direct assignment. |
| nodes/test/test_anomaly_detector.py | Scope rocketlib stub to the node import and restore afterward; switch to direct assignment. |
| nodes/test/guardrails/test_all.py | Scope rocketlib stub to dynamic module load and restore afterward. |
| nodes/test/cloud_tts/test_cloud_tts.py | Scope rocketlib/ai.* stubs to node import and restore afterward; switch to direct assignment. |
| nodes/test/atlas/test_isdeleted_filter.py | Scope pydantic stub to dynamic module load and restore afterward; switch to direct assignment. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@nodes/test/tool_mem0/test_tools.py`:
- Around line 143-147: Update the `_ensure_*` helpers defined above the
referenced blocks in `nodes/test/tool_mem0/test_tools.py` (lines 143-147) and
`nodes/test/tool_xtrace_memory/test_tools.py` (lines 134-138) to always create
fresh `types.ModuleType` stubs and assign them directly to `sys.modules`,
removing conditional reuse checks. Preserve each helper’s existing module names
and stub contents so the `_saved_core` snapshot/restore flow provides strict
isolation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 37f73ff6-4a3f-458d-9a17-7dc93b13d864
📒 Files selected for processing (8)
nodes/test/atlas/test_isdeleted_filter.pynodes/test/cloud_tts/test_cloud_tts.pynodes/test/guardrails/test_all.pynodes/test/test_anomaly_detector.pynodes/test/test_currency_convert_explicit.pynodes/test/tool_filesystem/test_read_size_cap.pynodes/test/tool_mem0/test_tools.pynodes/test/tool_xtrace_memory/test_tools.py
44da9d4 to
a8a705d
Compare
a8a705d to
561b75b
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@nodes/test/test_anomaly_detector.py`:
- Around line 13-26: Add the Ruff suppression comment `# noqa: E402` to the
module-level `from detector import AnomalyDetector` statement, leaving the
rocketlib stubbing and import setup unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: ca6c0c26-0363-4339-8e37-a6e6603d2f7d
📒 Files selected for processing (9)
nodes/test/atlas/test_isdeleted_filter.pynodes/test/cloud_tts/test_cloud_tts.pynodes/test/guardrails/test_all.pynodes/test/test_anomaly_detector.pynodes/test/test_currency_convert_explicit.pynodes/test/test_db_hydradb.pynodes/test/tool_filesystem/test_read_size_cap.pynodes/test/tool_mem0/test_tools.pynodes/test/tool_xtrace_memory/test_tools.py
Node tests stub rocketlib/pydantic/ai.common/requests/tenacity in sys.modules to import a node without the engine. Several did not restore them, so a stub leaked into the whole pytest session and broke unrelated tests (tool_slack, rerank_cohere, face_detection, ...) whenever collection order changed — the scenario in #1640, which surfaces in a plain `pytest nodes/test` run where the real modules are not loaded. - Wrap every stub in save/restore of sys.modules; for tests that build a self-contained rocketlib (tool_mem0, tool_xtrace_memory) assign fresh ModuleType stubs unconditionally so a foreign stub is never read or mutated. atlas drops the pydantic stub entirely (atlas.py loads the real rocketlib via ai.common.store, which needs real pydantic). Also covers cloud_tts, guardrails, ocr, tool_filesystem, qdrant, test_db_hydradb, anomaly_detector, currency_convert. - Add an autouse guard in conftest.py that fails any test leaving a stub core module in sys.modules, so the suite refuses to regress instead of leaking silently. Fixes #1640
561b75b to
f8bac23
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
nodes/test/test_db_hydradb.py:170
- The fixture cleanup runs after the yield, but if
_install_stubs()or_load_package()raises during setup, teardown won’t run and the partially-installedsys.modulesstubs can still leak into later tests. Wrapping the setup + yield in atry/finallyensures the restore always happens.
_saved = {name: sys.modules.get(name) for name in _managed}
Config = _install_stubs()
client_mod, iglobal_mod, iinstance_mod = _load_package()
yield SimpleNamespace(Config=Config, client_mod=client_mod, iglobal_mod=iglobal_mod, iinstance_mod=iinstance_mod)
14aa1b2 to
8a0b609
Compare
…own test) asclearuc pointed out the module-scoped guard was fundamentally wrong: stubs leak at IMPORT time (collection), long before any teardown, so a teardown check blamed whichever module happened to finish while a later module's leak was already in sys.modules — one leaker cascaded into N false failures naming innocent, already-fixed files. Move the guard to collection-time hooks (pytest_collectstart snapshots each module's guarded entries, pytest_collectreport compares right after import), undo each detected leak so it can't cascade, and fail at session finish (raising inside a collection hook gives INTERNALERROR). The stub heuristic now keys off __file__/__path__ so it also catches stubs built via module_from_spec. Extract the guard to _sys_modules_guard.py and add test_sys_modules_guard.py: it asserts the guard names the real leaker, undoes the leak, does not blame an earlier clean module, and detects a __spec__-carrying stub.
8a0b609 to
51703eb
Compare
049952d to
51703eb
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
nodes/test/_sys_modules_guard.py:118
pytest_sessionfinishforcessession.exitstatus = 1whenever leaks are present, which can mask more severe/diagnostic pytest exit codes (e.g. INTERNALERROR/usage errors) by downgrading them to1. Only override the exit status when the session was otherwise going to succeed (exitstatus == 0).
def pytest_sessionfinish(session, exitstatus): # noqa: ARG001
"""Fail the run when any test module leaked a core-module stub."""
if _core_leaks:
session.exitstatus = 1
asclearuc
left a comment
There was a problem hiding this comment.
Rechecked at 51703eb3. The blocking issue from the last round is fully resolved — this is an Approve from me.
The guard now runs at collection time (pytest_collectstart snapshot + pytest_collectreport compare) in a dedicated _sys_modules_guard.py, records leaks and fails the run in pytest_sessionfinish, and undoes each leak so one offender can't cascade. The detection helper is extracted as check_after_import and unit-tested. _is_stub_module now also handles the module_from_spec case (via __path__), and test_sys_modules_guard.py covers it explicitly — both things I'd flagged.
I re-ran the same experiment that failed last round: one deliberate import-time leaker among five modules that restore correctly.
- Before: 6 errors, five of them blaming innocent files.
- Now: the leak is attributed to the leaker alone, no false positives, and the run exits
1. Removing the leaker → exit0. The touched set is 169 passed; the guard's own 4 tests pass.
Two small follow-ups remain (both non-blocking, both flagged inline — fine to defer):
test_anomaly_detector.py/test_currency_convert_explicit.pystill leavesys.path.insert(...)and the genericsys.modules['detector']/['convert']entries for the session.- The hand-rolled save/restore is now in 8 files plus the guard;
patch.dict(sys.modules, {...})would collapse each to one line, and it's already the idiom here (tool_git,milvus,text_output).
Nice work turning the guard around, @dsapandora — the collection-time version plus its own test is exactly right. Approving.
Now that #1642's collection-time guard undoes the sys.modules stub leaks, the reordered pytest collection is harmless, so the test folders can follow their nodes: nodes/test/{astra_db,atlas,chroma,milvus,pinecone,qdrant,weaviate} -> store_*. Tests load src via ../../src/nodes/store_* (unchanged depth), so only the two ocr breadcrumbs and two stale "Usage" docstrings needed the new paths.
…o graph_* (#1636) * refactor(nodes): rename vector stores to store_* and graph-first db to graph_* Rename the nine vector-store nodes to the store_ prefix (matching their `store` classType) and the two graph-first databases (ArangoDB, HydraDB) to graph_. Update each services.json path to the new python module and add graph to the ArangoDB/HydraDB classType (database and tool kept). protocol and prefix are left untouched, so saved pipes keep resolving; no services-version bump. Docs redirects map the old /nodes/<old> URLs to the new ones. Test module imports and path references follow the renamed src folders. constraints.lock and docs static pages are regenerated by CI. Node test folders keep their original names so pytest collection order is unchanged. * docs(nodes): align renamed node docs with store_*/graph_* identity Co-located-docs follow-through for the rename in this PR (per asclearuc review): - 10 node READMEs still carried the old title and a "View source" link to the old folder (# milvus -> # store_milvus, .../nodes/milvus -> .../nodes/store_milvus), which 404s after merge. store_postgres was already done. - docs/README-nodes.md: the Vector Databases & Stores table listed the old folder-derived ids; refreshed to store_*. The catalog is hand-maintained (no generator), so the misleading "generated" note was corrected to match. - tools/dependabot-smoke/smoke-nodes.sh: index_search -> store_elasticsearch so the opensearchpy smoke check keeps resolving after the folder rename. * test(nodes): rename test folders to store_* to match the renamed nodes Now that #1642's collection-time guard undoes the sys.modules stub leaks, the reordered pytest collection is harmless, so the test folders can follow their nodes: nodes/test/{astra_db,atlas,chroma,milvus,pinecone,qdrant,weaviate} -> store_*. Tests load src via ../../src/nodes/store_* (unchanged depth), so only the two ocr breadcrumbs and two stale "Usage" docstrings needed the new paths.
…ests (rocketride-org#1642) * test(nodes): isolate core-module stubs in node tests + regression guard Node tests stub rocketlib/pydantic/ai.common/requests/tenacity in sys.modules to import a node without the engine. Several did not restore them, so a stub leaked into the whole pytest session and broke unrelated tests (tool_slack, rerank_cohere, face_detection, ...) whenever collection order changed — the scenario in rocketride-org#1640, which surfaces in a plain `pytest nodes/test` run where the real modules are not loaded. - Wrap every stub in save/restore of sys.modules; for tests that build a self-contained rocketlib (tool_mem0, tool_xtrace_memory) assign fresh ModuleType stubs unconditionally so a foreign stub is never read or mutated. atlas drops the pydantic stub entirely (atlas.py loads the real rocketlib via ai.common.store, which needs real pydantic). Also covers cloud_tts, guardrails, ocr, tool_filesystem, qdrant, test_db_hydradb, anomaly_detector, currency_convert. - Add an autouse guard in conftest.py that fails any test leaving a stub core module in sys.modules, so the suite refuses to regress instead of leaking silently. Fixes rocketride-org#1640 * Potential fix for pull request finding * Potential fix for pull request finding * fix(test): scope the sys.modules guard to module level The autouse guard ran per test, so it false-positived on tests that stub a core module in a scope='module' fixture and restore it at module teardown (e.g. db_clickhouse). Make the guard scope='module' so it checks after the module's own fixtures have torn down; it still catches import-time stubs that never restore. * test(nodes): address review — package stubs, hydradb try/finally, qdrant assign - Mark the ai/ai.common stubs as packages (__path__ = []) in tool_mem0, tool_xtrace_memory and cloud_tts so ai.common.* sub-imports resolve, matching the pattern already used in face_detection/ocr/graph tests. - test_db_hydradb: wrap fixture setup + yield + restore in try/finally so sys.modules is restored even if setup raises. - qdrant/test_search_threshold: replace setdefault with direct assignment. - test_anomaly_detector: add noqa: E402 to the post-stub import. * fix(test): guard sys.modules at collection time, not teardown (+ its own test) asclearuc pointed out the module-scoped guard was fundamentally wrong: stubs leak at IMPORT time (collection), long before any teardown, so a teardown check blamed whichever module happened to finish while a later module's leak was already in sys.modules — one leaker cascaded into N false failures naming innocent, already-fixed files. Move the guard to collection-time hooks (pytest_collectstart snapshots each module's guarded entries, pytest_collectreport compares right after import), undo each detected leak so it can't cascade, and fail at session finish (raising inside a collection hook gives INTERNALERROR). The stub heuristic now keys off __file__/__path__ so it also catches stubs built via module_from_spec. Extract the guard to _sys_modules_guard.py and add test_sys_modules_guard.py: it asserts the guard names the real leaker, undoes the leak, does not blame an earlier clean module, and detects a __spec__-carrying stub.
…o graph_* (rocketride-org#1636) * refactor(nodes): rename vector stores to store_* and graph-first db to graph_* Rename the nine vector-store nodes to the store_ prefix (matching their `store` classType) and the two graph-first databases (ArangoDB, HydraDB) to graph_. Update each services.json path to the new python module and add graph to the ArangoDB/HydraDB classType (database and tool kept). protocol and prefix are left untouched, so saved pipes keep resolving; no services-version bump. Docs redirects map the old /nodes/<old> URLs to the new ones. Test module imports and path references follow the renamed src folders. constraints.lock and docs static pages are regenerated by CI. Node test folders keep their original names so pytest collection order is unchanged. * docs(nodes): align renamed node docs with store_*/graph_* identity Co-located-docs follow-through for the rename in this PR (per asclearuc review): - 10 node READMEs still carried the old title and a "View source" link to the old folder (# milvus -> # store_milvus, .../nodes/milvus -> .../nodes/store_milvus), which 404s after merge. store_postgres was already done. - docs/README-nodes.md: the Vector Databases & Stores table listed the old folder-derived ids; refreshed to store_*. The catalog is hand-maintained (no generator), so the misleading "generated" note was corrected to match. - tools/dependabot-smoke/smoke-nodes.sh: index_search -> store_elasticsearch so the opensearchpy smoke check keeps resolving after the folder rename. * test(nodes): rename test folders to store_* to match the renamed nodes Now that rocketride-org#1642's collection-time guard undoes the sys.modules stub leaks, the reordered pytest collection is harmless, so the test folders can follow their nodes: nodes/test/{astra_db,atlas,chroma,milvus,pinecone,qdrant,weaviate} -> store_*. Tests load src via ../../src/nodes/store_* (unchanged depth), so only the two ocr breadcrumbs and two stale "Usage" docstrings needed the new paths.
Summary
rocketlib+ submodules,pydantic, and theirai.common.*/requests/tenacitycompanions) intosys.modulesat import time and never restored them. The stub stayed global for the rest of the pytest session and leaked into unrelated tests that import the real module.sys.modulesin afinally. Switchsetdefaultto direct assignment so a stub left contaminated by another test can't be reused.Type
test / bug fix (test isolation)
Testing
./builder testpasses (runs in CI on this PR)Checklist
Linked Issue
Fixes #1640
Context: this unblocks #1636 (node-family renames), which does nothing to these tests but reorders pytest collection and surfaces the leak. With isolation fixed here, any reorder stays green.
Summary by CodeRabbit
sys.modulesstate afterward.