fix(pyamber, 1.1): make ExecutorManager module names process-globally unique#4868
Merged
Yicong-Huang merged 1 commit intoMay 3, 2026
Conversation
…ue (apache#4717) Lift `ExecutorManager`'s tmp module-name counter from a per-instance attribute to a process-wide `itertools.count(1)`. Generated module names (`udf-vN`) are now unique across every instance in the same Python process, so the `if module_name in sys.modules: clear()+reload()` recovery branch (which silently returned a stale class on 3.11) is unreachable and removed. Production path is unchanged — a worker process holds exactly one `ExecutorManager` and its counter was already monotonic across reconfigures. Closes apache#4705. `sbt 'WorkflowExecutionService / Test / testOnly test_executor_manager test_main_loop'` — 23/23 pass. The combined run that previously leaked `TestOperator` from `test_executor_manager.py` into `test_main_loop.py`'s fixtures is now clean. `test_executor_manager.py` updated for the new semantics: assertions on `executor_version == 0/1` and `operator_module_name == "udf-v1"` change to relative-ordering checks since the counter's absolute starting value depends on prior tests in the session. Generated-by: Claude Code (Opus 4.7, 1M context)
Yicong-Huang
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## release/v1.1.0-incubating #4868 +/- ##
===============================================================
+ Coverage 41.48% 41.50% +0.01%
Complexity 1984 1984
===============================================================
Files 952 952
Lines 33801 33796 -5
Branches 3713 3713
===============================================================
+ Hits 14024 14026 +2
+ Misses 19011 19003 -8
- Partials 766 767 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Yicong-Huang
changed the title
bobbai00
approved these changes
May 3, 2026
Yicong-Huang
merged commit 22f1d0f
into
apache:release/v1.1.0-incubating
27 of 32 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Backport #4717 (commit
6ae0c46312ca744b4f761c88f1ec172ba0d41d13onmain) ontorelease/v1.1.0-incubating.ExecutorManagerpreviously used a per-instanceexecutor_versioncounter for theudf-vNtmp module name, so a freshExecutorManageralways producedudf-v1and collided with whateverudf-v1was already cached insys.modulesfrom an earlier instance. The post-collisionclear()+importlib.reload()recovery silently returned a stale class on Python 3.11. Lift the counter to a class-levelitertools.count(1)so module names are unique across every instance in the same Python process; the recovery branch becomes unreachable and is removed.This unblocks #4636 (and any future
release/*-labelled PR): the auto-backport leg'score/runnables/test_main_loop.py::test_batch_dp_thread_can_process_batchwas failing on the 3.11 matrix entry against this branch withAttributeError: 'TestOperator' object has no attribute 'count'(the stale-class symptom), and the failed test left a non-daemonmain_loop_threadalive that prevented pytest from exiting — surfacing as a 30+ minute hang.Any related issues, documentation, discussions?
Backports #4717. Original issue: #4705.
How was this PR tested?
Cherry-pick of an already-reviewed and merged commit. One conflict in
test_executor_manager.py(release branch lacked the trailingTestUpdateExecutortest class that #4717 introduced); resolved by taking the incoming version verbatim. Local syntax +ruff format --checkpass on both modified files. CI on this PR will exercise the change against the release branch's full Python matrix.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-7)