enable default algo template caching?

[wyli]

currently the default file is downloaded to a temporary folder and therefore skips any caching

MONAI/monai/apps/auto3dseg/bundle_gen.py

Lines 317 to 320 in cd2f4e1

	zip_download_dir = TemporaryDirectory()
	algo_compressed_file = os.path.join(zip_download_dir.name, "algo_templates.tar.gz")
	download_and_extract(default_algo_zip if algos is None else algos, algo_compressed_file, algo_path)
	zip_download_dir.cleanup()

this causes an issue when running multiple tests:

[2022-10-18T08:02:43.002Z] ======================================================================
[2022-10-18T08:02:43.002Z] ERROR: test_autorunner_hpo (__main__.TestAutoRunner)
[2022-10-18T08:02:43.002Z] ----------------------------------------------------------------------
[2022-10-18T08:02:43.002Z] Traceback (most recent call last):
[2022-10-18T08:02:43.002Z]   File "tests/test_integration_autorunner.py", line 136, in test_autorunner_hpo
[2022-10-18T08:02:43.002Z]     runner.run()
[2022-10-18T08:02:43.002Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/auto_runner.py", line 568, in run
[2022-10-18T08:02:43.002Z]     bundle_generator = BundleGen(
[2022-10-18T08:02:43.002Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 319, in __init__
[2022-10-18T08:02:43.002Z]     download_and_extract(default_algo_zip if algos is None else algos, algo_compressed_file, algo_path)
[2022-10-18T08:02:43.002Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/utils.py", line 312, in download_and_extract
[2022-10-18T08:02:43.002Z]     download_url(url=url, filepath=filename, hash_val=hash_val, hash_type=hash_type, progress=progress)
[2022-10-18T08:02:43.002Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/utils.py", line 203, in download_url
[2022-10-18T08:02:43.002Z]     _download_with_progress(url, tmp_name, progress=progress)
[2022-10-18T08:02:43.002Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/utils.py", line 114, in _download_with_progress
[2022-10-18T08:02:43.002Z]     raise e
[2022-10-18T08:02:43.002Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/utils.py", line 107, in _download_with_progress
[2022-10-18T08:02:43.002Z]     urlretrieve(url, filepath, reporthook=t.update_to)
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 247, in urlretrieve
[2022-10-18T08:02:43.002Z]     with contextlib.closing(urlopen(url, data)) as fp:
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
[2022-10-18T08:02:43.002Z]     return opener.open(url, data, timeout)
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 531, in open
[2022-10-18T08:02:43.002Z]     response = meth(req, response)
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
[2022-10-18T08:02:43.002Z]     response = self.parent.error(
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 563, in error
[2022-10-18T08:02:43.002Z]     result = self._call_chain(*args)
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
[2022-10-18T08:02:43.002Z]     result = func(*args)
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 755, in http_error_302
[2022-10-18T08:02:43.002Z]     return self.parent.open(new, timeout=req.timeout)
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 531, in open
[2022-10-18T08:02:43.002Z]     response = meth(req, response)
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
[2022-10-18T08:02:43.002Z]     response = self.parent.error(
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 569, in error
[2022-10-18T08:02:43.002Z]     return self._call_chain(*args)
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
[2022-10-18T08:02:43.002Z]     result = func(*args)
[2022-10-18T08:02:43.002Z]   File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
[2022-10-18T08:02:43.002Z]     raise HTTPError(req.full_url, code, msg, hdrs, fp)
[2022-10-18T08:02:43.002Z] urllib.error.HTTPError: HTTP Error 503: Egress is over the account limit.
[2022-10-18T08:02:43.002Z] 
[2022-10-18T08:02:43.002Z] ----------------------------------------------------------------------

an option of caching the default algo could avoid the repeated donwloading cc @Nic-Ma @mingxin-zheng

[mingxin-zheng]

Thanks @wyli for catching it. Does this happen when two or more instances are downloading the same package?

[wyli]

yes I think so, my impression is that it's happening during concurrent downloadings from the same IP...

[mingxin-zheng]

Okay, if we're downloading the algo_template.tar.gz to a set cache folder, then concurrent downloading processes may compete with each other on the file I/O. How about using a try to catch the urllib.error.HTTPError ?

[wyli]

I haven't looked into a better idea, but previously in the mmar utilities we download the zip file to the torch hub caching folder, and unzip to a user-specified folder:

MONAI/monai/apps/mmars/mmars.py

Lines 150 to 151 in 8aafd88

	filepath=mmar_dir / f'{v["name"]}_{ver}.zip',
	output_dir=model_dir,

the torch hub caching folder is by default ~/.cache/torch and it's cached during CI:

MONAI/.github/workflows/setupapp.yml

Lines 38 to 39 in 4bbf125

	~/.cache/pip
	~/.cache/torch

also, maybe file IO is not a critical issue because the downloading is to a temporary location first and then moved to the destination

MONAI/monai/apps/utils.py

Line 211 in 8aafd88

shutil.move(f"{tmp_name}", f"{filepath}") # copy the downloaded to a user-specified cache.

[mingxin-zheng]

@wyli I think after #5623 BundleGen will make 3 attempts to download the templates to resolve the HTTP errors. Can we close this issue?