Skip to content

test(nodes): keep the real depends and engLib in sys.modules for the whole test run (#3097) - #3098

Merged
asclearuc merged 2 commits into
developfrom
fix/RR-3097-depends-stub-leak
Nov 7, 2026
Merged

asclearuc merged 2 commits into
developfrom
fix/RR-3097-depends-stub-leak

Conversation

@asclearuc

@asclearuc asclearuc commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • In a run that collects all of nodes/test, depends() installed nothing during the tests. tool_git/test_tools.py put a stub depends and a MagicMock engLib in sys.modules when pytest imported it, and left them there for the rest of the session. pytest imports every test module before the first test runs, so:
    • every later depends() / load_depends() call did nothing. On an engine without torch, test_embedding_image.py failed with No module named 'torch', also with the command its docstring gives (builder nodes:test-full --pytest="-m skip_node -k embedding_image"), which collects all of nodes/test too. Run on its own, the test installs torch first;
    • a module that first imports engLib later in the run (ai.common.ssl, ai.node, depends.py) got the MagicMock.
  • tool_git/test_tools.py now puts both stand-ins in sys.modules only while tool_git loads, inside the patch.dict block that already covers that import. git_repo.py calls depends() only at import, so nothing in these tests needs them afterwards.
  • The sys.modules guard (nodes/test/_sys_modules_guard.py) pins depends and engLib, as it pins numpy since test(nodes): keep the real numpy in sys.modules for the whole test run (#3019, #3025) #3068: it checks them at collection time and after every test, a test that leaves either replaced or removed fails, and the real module is put back.
    • engLib is compiled into the engine, so it has no __file__, and the guard's stub check (_is_stub_module) took it for a stub; it could not have been pinned. A module loaded by BuiltinImporter now counts as real. A hand-built module with the same name is still a stub.
  • Why not pytest.importorskip('torch') in test_embedding_image.py: on an engine without torch the test would be skipped forever instead of installing torch and running, and every other first-use install in the run would stay a silent no-op.

Type

fix (tests only)

Testing

  • Tests added or updated
  • Tested locally
  • ./builder test passes

Local runs with ./dist/server/engine.exe -m pytest ... on a fresh light dist (no torch at the start):

Check develop this branch
nodes/test -m skip_node -k embedding_image (the docstring's command, run directly) 1 failed: No module named 'torch' 1 passed; depends() installed torch 2.10.0+cu128 and transformers 4.53.3 first (4 min 17 s)
sys.modules['depends'] after collecting test_embedding_image.py and tool_git/test_tools.py in one session module with no __file__ .../rocketlib-python/lib/depends.py
Whole nodes/test in one process (-m "not skip_node", no *_full, as builder nodes:test) 8637 passed, 535 skipped 8644 passed (the 7 new guard tests), 535 skipped, no leak report
Same run with the guard pinning depends, before the tool_git fix — the guard names one leak: tool_git/test_tools.py left ['depends']; no other module or test
Same run with the final code (depends and engLib pinned, tool_git fixed) — no leak report; with the tool_git fix reverted, the guard names tool_git/test_tools.py for depends and, in a separate run, for engLib
Collect-only of all 9235 tests, skip_node and *_full included — no leak report
Affected suites (guard, tool_git, every suite that stubs depends or engLib) with -n 3, and in reversed order — 1832 passed each
Mutations: tool_git leaves depends / engLib for the session; built-in modules count as stubs; depends, engLib or both not pinned — each one fails the run

With the real depends() active during the tests, the whole-nodes/test run resolved the requirements of rerank_cohere, store_chroma and store_weaviate, which their tests load while they run, and installed 7 packages (authlib, cohere, fastavro, filelock, grpcio-health-checking, tokenizers, types-requests). On develop those calls did nothing. The run was not slower (2 min 42 s against 3 min 23 s on develop).

ruff check and ruff format --check (0.16.5) pass on the changed files.

Self-review checklist items checked: 3 (every path to a stub depends / engLib: module level, setdefault, per test, patch.dict, attributes of the real module), 4 (the guard tests call the real hooks; the docstring's command run end to end), 7 and 11 (path table for _is_stub_module, a test per changed row), 13 (each changed file run with the whole nodes/test in one process), 21 and 26c (mutations show no test passes for the wrong reason), 29 (every guard test puts sys.modules back on every path), 30 and 34 (docstring and nesting tools).

Known limit: the per-test check runs only for tests that run; the *_full and skip_node tests were collected, not run, here (they need GPUs, models or databases).

Checklist

  • Commit messages follow conventional commits
  • No secrets or credentials included
  • Wiki updated (if applicable)
  • Breaking changes documented (if applicable)

Linked Issue

Fixes #3097

Summary by CodeRabbit

  • Tests
    • Improved test isolation by limiting temporary substitutes for required engine components to the tests that need them and restoring previous state afterward.
    • Added checks to catch unexpected changes to protected components and to distinguish genuine interpreter-loaded modules from test substitutes.
    • Documented that the image-embedding test fails when PyTorch is unavailable; this release does not resolve that failure.

@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The test guard now pins depends and engLib, treats built-in modules as real, and detects replacements. The tool_git tests scope their import stubs and restore the prior sys.modules entries.

Changes: Module guard and test isolation

Layer / File(s) Summary
Pin engine modules and verify guard behavior
nodes/test/_sys_modules_guard.py, nodes/test/test_sys_modules_guard.py
The guard recognizes modules loaded by BuiltinImporter as real and pins depends and engLib. Tests cover built-in module classification, import-time replacements, teardown failures, and restoration of pinned modules.
Scope tool_git import stubs
nodes/test/tool_git/test_tools.py, CHANGELOG.md
The tool_git tests provide engLib and depends stubs during git_repo.py import. The changelog records these changes and a reported torch-related test failure.

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix · Severity of issue fixed: Low

Possibly related PRs

Merge Risk: Minimal · up to 84a97

This change scopes test stubs and strengthens the module guard in the nodes test suite. No merge-blocking risk was identified.

Pre-merge checks: Passed 5

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title describes the main change: preserving the real depends and engLib modules during the nodes test run.
Linked Issues check Passed [#3097] tool_git/test_tools.py limits the depends and engLib stubs to the import. The guard detects and restores replaced or removed modules. Automated tests cover the guard and built-in module …
Out of Scope Changes check Passed [#3097] Stub isolation, the guard, tests, and changelog support the test isolation fix. The engLib guard addresses a related leak from the same test setup.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Full details: Linked Issues check

Explanation

[#3097] tool_git/test_tools.py limits the depends and engLib stubs to the import. The guard detects and restores replaced or removed modules. Automated tests cover the guard and built-in module recognition. The PR reports that the affected embedding test installed torch and passed.

Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 3 files. (1 skipped: 1 unsupported.)

Finishing Touches

Generate docstrings

  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 9475fe7 and 84a97d8.

Files selected for processing: 4

  • CHANGELOG.md
  • nodes/test/_sys_modules_guard.py
  • nodes/test/test_sys_modules_guard.py
  • nodes/test/tool_git/test_tools.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor
🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID and forum thread ID. Do not edit or delete.

@github-actions github-actions Bot added docs Documentation module:nodes Python pipeline nodes tests labels Nov 6, 2026
…whole test run (#3097)

- `tool_git/test_tools.py` puts its `depends` and `engLib` stand-ins in `sys.modules` only while `tool_git` loads
- the `sys.modules` guard pins `depends` and `engLib` the way it pins numpy
- a module loaded by `BuiltinImporter` (such as `engLib`) counts as real, not as a stub
- guard tests for both modules

@dk-rocketride dk-rocketride left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. The stubs now live only for the tool_git import, inside the patch.dict that already covered it, and pinning depends and engLib makes the next leak fail in teardown instead of silently turning depends() into a no-op. Outside the engine _import_pinned skips modules that don't import, so plain-python runs are unaffected. The BuiltinImporter check has its own test. CI is green on all three platforms.

@asclearuc
asclearuc merged commit cc55754 into develop Nov 7, 2026
@asclearuc
asclearuc deleted the fix/RR-3097-depends-stub-leak branch November 7, 2026 15:49
Rod-Christensen added a commit that referenced this pull request Nov 9, 2026
Brings in the first ten develop PRs since base 7e009b1. Every PR's intent was
checked on the merged tree; the two files that conflicted were resolved by
re-expressing develop's behaviour in this branch's shape, not by taking a side.

PRs and how each intent survives:
- #3062 feat(appdev,shell): point failed builds to their build log - preserved
  (TS only, no overlap).
- #3081 test(client-typescript): more time for slow-runner startup tests -
  preserved (tests only).
- #3055 fix(nodes): store_milvus writes and reads a real Milvus with pymilvus 3 -
  preserved: develop's rewrite merged cleanly onto the converted driver
  (self.config shape untouched by the PR); store_milvus suite + mock + shapes
  tests pass.
- #3069 feat(client): invite() returning the invited member in both SDKs -
  preserved; client-python tests/test_account.py 3 passed; docs merged.
- #2965 fix(ai): generated SQL on a read-only, time-limited connection -
  preserved; packages/ai/tests/ai/common/database 392 passed (live DSN tests
  skipped).
- #3061 docs(cli): per-CLI command tables + CI drift check - preserved;
  tests/test_validate_cli_docs.py 24 passed.
- #3096 fix(appdev,shell): review follow-up on build-log change - preserved
  (TS only).
- #3091 fix(mcp): credential fields opt out of automatic wiring - RE-EXPRESSED.
  develop added a catalog-file option `auto_wire: false`; this branch has no
  catalog file (credentials.json and nodes/scripts/gen-credentials.mjs are
  deleted, the catalog is derived from each node's resolved service
  definition). The same rule now lives in the definition walk: a credential
  found inside an array item (a list-row field the node reads per row) is
  described but `auto_wire=False` and `required=False`, and `evaluate()` only
  wires `auto_wire` fields. develop's six-case test is kept as
  test_array_row_credential_is_described_but_never_required_or_wired against
  catalog_from_definitions. gen-credentials.mjs stays deleted (superseded).
- #3098 test(nodes): keep real depends and engLib in sys.modules - preserved;
  test_sys_modules_guard.py + tool_git tests pass.
- #3076 docs(vscode): connection modes - preserved (docs only).

Verification on the merged tree: ruff format/check clean on the touched .py;
grep gate for getNodeConfig/getMultiProviderConfig/overlay_top_level/
_warnMisnamedKeys/preconfig over packages/ai/src, rocketlib-python, nodes/src
= 0; old-format services*.json = tool_pipedrive enum label only;
nodes/test/test_import_gate.py + store_milvus + guard + tool_git = 328 passed;
mcp credentials/integrations + database + device budget = 392 passed;
./builder nodes:sync ok.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation module:nodes Python pipeline nodes tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nodes tests: a run of the whole nodes/test installs nothing on first use, so test_embedding_image fails with No module named 'torch'

2 participants