AutoRunner enhancements: local folder and algorithm subsets

[myron]

AutoRunner class improvements

• adds auto-runner option templates_path_or_url to optionally accept a local templates folder location. The default (None) is the same: to download the release zip e.g. AutoRunner(templates_path_or_url='/my/local/algorithm_templates')

• enhances algos input option to define a subset of algorithm names to run (e.g. AutoRunner(algos=['segresnet','dints'])

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[myron]

the windows tests are failing again , for some unrelated reason...

[Nic-Ma]

I see the flake8 tests didn't pass, could you please help fix it?

Thanks.

[mingxin-zheng]

I checked out the branch and run the integration test for AutoRunner. Some failures need to be fixed:

(base) root@adminn-MS-7D31:/workspace/monai/monai-in-dev# python tests/test_integration_autorunner.py
2022-12-01 15:16:54,155 - INFO - AutoRunner using work directory /tmp/tmplf3777to/work_dir
2022-12-01 15:16:54,156 - INFO - Loading input config /tmp/tmplf3777to/data_src_cfg.yaml
2022-12-01 15:16:54,157 - INFO - The output_dir is not specified. /tmp/tmplf3777to/work_dir/ensemble_output will be used to save ensemble predictions
2022-12-01 15:16:54,157 - INFO - Directory /tmp/tmplf3777to/work_dir/ensemble_output is created to save ensemble predictions
2022-12-01 15:16:54,157 - INFO - Running data analysis...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 45.86it/s]
algo_templates.tar.gz: 296kB [00:01, 243kB/s]                                                                                                                                          
2022-12-01 15:16:55,743 - INFO - Downloaded: /tmp/tmpphes5ts9/algo_templates.tar.gz
2022-12-01 15:16:55,744 - INFO - Expected md5 is None, skip md5 check for file /tmp/tmpphes5ts9/algo_templates.tar.gz.
2022-12-01 15:16:55,744 - INFO - Writing into directory: /tmp/tmplf3777to/work_dir.
E2022-12-01 15:16:55,897 - INFO - AutoRunner using work directory /tmp/tmpvo558ukw/work_dir
2022-12-01 15:16:55,898 - INFO - Loading input config /tmp/tmpvo558ukw/data_src_cfg.yaml
2022-12-01 15:16:55,898 - INFO - The output_dir is not specified. /tmp/tmpvo558ukw/work_dir/ensemble_output will be used to save ensemble predictions
2022-12-01 15:16:55,898 - INFO - Directory /tmp/tmpvo558ukw/work_dir/ensemble_output is created to save ensemble predictions
2022-12-01 15:16:55,898 - INFO - Running data analysis...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 75.03it/s]
algo_templates.tar.gz: 296kB [00:01, 298kB/s]                                                                                                                                          
2022-12-01 15:16:57,142 - INFO - Downloaded: /tmp/tmp2m08_yi6/algo_templates.tar.gz
2022-12-01 15:16:57,142 - INFO - Expected md5 is None, skip md5 check for file /tmp/tmp2m08_yi6/algo_templates.tar.gz.
2022-12-01 15:16:57,143 - INFO - Writing into directory: /tmp/tmpvo558ukw/work_dir.
E
======================================================================
ERROR: test_autorunner (__main__.TestAutoRunner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_integration_autorunner.py", line 108, in test_autorunner
    runner.run()
  File "/workspace/monai/monai-in-dev/monai/apps/auto3dseg/auto_runner.py", line 566, in run
    bundle_generator = BundleGen(
  File "/workspace/monai/monai-in-dev/monai/apps/auto3dseg/bundle_gen.py", line 349, in __init__
    for name in algos:
TypeError: 'NoneType' object is not iterable

======================================================================
ERROR: test_autorunner_hpo (__main__.TestAutoRunner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_integration_autorunner.py", line 138, in test_autorunner_hpo
    runner.run()
  File "/workspace/monai/monai-in-dev/monai/apps/auto3dseg/auto_runner.py", line 566, in run
    bundle_generator = BundleGen(
  File "/workspace/monai/monai-in-dev/monai/apps/auto3dseg/bundle_gen.py", line 349, in __init__
    for name in algos:
TypeError: 'NoneType' object is not iterable

----------------------------------------------------------------------
Ran 2 tests in 3.141s

FAILED (errors=2)

[myron]

I see the flake8 tests didn't pass, could you please help fix it?

Thanks.

done

[myron]

I checked out the branch and run the integration test for AutoRunner. Some failures need to be fixed:

done

[wyli]

thanks, I put some some comments inline

[myron]

@wyli @mingxin-zheng I've made the requested changes, plz take a look.

[wyli]

looks good to me, please confirm @mingxin-zheng

[mingxin-zheng]

Hi @myron , I just had a small comment and everything else looked good! Thx

[wyli]

/build