Running mobie-utils-python on Windows

[K-Meech]

@constantinpape I'm trying out your conda package on Windows and having a few issues.
It installs fine from conda, but when I import mobie it errors with the error below. Any ideas?:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\meechan\AppData\Local\Continuum\anaconda3\envs\mobie-utils-python\lib\site-packages\mobie\__init__.py", line 1, in <module>
    from .image_data import add_image_data
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\meechan\AppData\Local\Continuum\anaconda3\envs\mobie-utils-python\lib\site-packages\mobie\image_data.py", line 6, in <module>
    from mobie.import_data import import_raw_volume
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\meechan\AppData\Local\Continuum\anaconda3\envs\mobie-utils-python\lib\site-packages\mobie\import_data\__init__.py", line 1, in <module>
    from .from_node_labels import import_segmentation_from_node_labels
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\meechan\AppData\Local\Continuum\anaconda3\envs\mobie-utils-python\lib\site-packages\mobie\import_data\from_node_labels.py", line 5, in <module>
    from cluster_tools.write import WriteLocal, WriteSlurm
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\meechan\AppData\Local\Continuum\anaconda3\envs\mobie-utils-python\lib\site-packages\cluster_tools\__init__.py", line 1, in <module>
    from .workflows import MulticutSegmentationWorkflow
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\meechan\AppData\Local\Continuum\anaconda3\envs\mobie-utils-python\lib\site-packages\cluster_tools\workflows.py", line 6, in <module>
    from .graph import GraphWorkflow
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\meechan\AppData\Local\Continuum\anaconda3\envs\mobie-utils-python\lib\site-packages\cluster_tools\graph\__init__.py", line 1, in <module>
    from .graph_workflow import GraphWorkflow
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\meechan\AppData\Local\Continuum\anaconda3\envs\mobie-utils-python\lib\site-packages\cluster_tools\graph\graph_workflow.py", line 4, in <module>
    from . import initial_sub_graphs as initial_tasks
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\meechan\AppData\Local\Continuum\anaconda3\envs\mobie-utils-python\lib\site-packages\cluster_tools\graph\initial_sub_graphs.py", line 9, in <module>
    import nifty.distributed as ndist
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\meechan\AppData\Local\Continuum\anaconda3\envs\mobie-utils-python\lib\site-packages\nifty\distributed\__init__.py", line 1, in <module>
    from . import _distributed as __distributed
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ImportError: dynamic module does not define module export function (PyInit__distributed)

[constantinpape]

Thanks for checking this out!
I know what the issue is. I will try to fix it and let you know once I think it's fixed.

[constantinpape]

Ok, for a clean solution I would need to fix the issues encoutered in this PR: conda-forge/nifty-feedstock#3
There is a faster but more hacky solution, so I would like to see if I can fix this (probably need to ask some conda-forge people for help).
@K-Meech do you need this on Windows right now or were you just testing it out?

[K-Meech]

I was trying to make a mobie project with the zebrafish data (converting segmentations to n5 via fiji is still buggy). So no massive rush - if you have time, I can just send you the link to the data and you can run it?

[constantinpape]

So no massive rush - if you have time, I can just send you the link to the data and you can run it?

Sounds good, just send me the link :).

[constantinpape]

@K-Meech I think I managed to fix this issue now. You would need to update your conda environment and use one of the nifty windows packages that were updated today, see https://anaconda.org/conda-forge/nifty/files.
(This might not be so easy because this still has the same version number as the previous pakcages, I will try to upload a bumped version soon, so that I can fix it directly in the dependencies).

It would be great if you could give it a try next week.

[K-Meech]

Thanks @constantinpape! I updated my conda environment and re-ran.
Now it gets past import mobie but errors at a later step.

These are the parameters I'm running with:

import mobie

mobie_root = 'C:\\Users\\meechan\\Documents\\temp\\minimal_mobie_project\\data'
target = "local"
max_jobs = 1

membrane = "C:\\Users\\meechan\\Documents\\test_images\\zebrafish\\0B51F8B46C_8bit_lynEGFP.tif"

mobie.initialize_dataset(input_path=membrane, input_key="", root=mobie_root, dataset_name="zebrafish",
                         raw_name="membrane", resolution=(0.2245383, 0.0992103, 0.0992103),
                         chunks=(64, 64, 64), scale_factors=3 * [[2, 2, 2]], is_default=True,
                         target=target, max_jobs=max_jobs, unit="microns")

And this is the error:

DEBUG: Checking if DownscalingWorkflow(tmp_folder=tmp_zebrafish, max_jobs=1, config_dir=tmp_zebrafish\configs, target=local, dependency=DummyTask, input_path=C:\Users\meechan\Documents\test_images\zebrafish\0B51F8B46C_8bit_lynEGFP.tif, input_key=, scale_factors=[[2, 2, 2], [2, 2, 2], [2, 2, 2]], halos=[[2, 2, 2], [2, 2, 2], [2, 2, 2]], metadata_format=bdv.n5, metadata_dict={"resolution": [0.2245383, 0.0992103, 0.0992103], "unit": "microns"}, output_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, output_key_prefix=, force_copy=False, skip_existing_levels=False, scale_offset=0) is complete
C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\luigi\parameter.py:279: UserWarning: Parameter "dtype" with value "None" is not of type string.
  warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\luigi\parameter.py:279: UserWarning: Parameter "scale_factor" with value "(2, 2, 2)" is not of type string.
  warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if WriteDownscalingMetadata(tmp_folder=tmp_zebrafish, output_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, scale_factors=[[2, 2, 2], [2, 2, 2], [2, 2, 2]], dependency=DownscalingLocal, metadata_format=bdv.n5, metadata_dict={"resolution": [0.2245383, 0.0992103, 0.0992103], "unit": "microns"}, output_key_prefix=, scale_offset=0, prefix=downscaling) is complete
INFO: Informed scheduler that task   DownscalingWorkflow_tmp_zebrafish_co_DummyTask_False_b6a307b20a   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_zebrafish, max_jobs=1, config_dir=tmp_zebrafish\configs, input_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, input_key=setup0/timepoint0/s2, output_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, output_key=setup0/timepoint0/s3, scale_factor=(2, 2, 2), scale_prefix=s3, halo=[2, 2, 2], effective_scale_factor=[8, 8, 8], dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task   WriteDownscalingMetadata_DownscalingLocal___resolution_____bdv_n5_42789540b8   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_zebrafish, max_jobs=1, config_dir=tmp_zebrafish\configs, input_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, input_key=setup0/timepoint0/s1, output_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, output_key=setup0/timepoint0/s2, scale_factor=(2, 2, 2), scale_prefix=s2, halo=[2, 2, 2], effective_scale_factor=[4, 4, 4], dependency=DownscalingLocal) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_zebrafish_co_DownscalingLocal__8__8__8__df1991e9d6   has status   PENDING
DEBUG: Checking if DownscalingLocal(tmp_folder=tmp_zebrafish, max_jobs=1, config_dir=tmp_zebrafish\configs, input_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, input_key=setup0/timepoint0/s0, output_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, output_key=setup0/timepoint0/s1, scale_factor=(2, 2, 2), scale_prefix=s1, halo=[2, 2, 2], effective_scale_factor=[2, 2, 2], dependency=CopyVolumeLocal) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_zebrafish_co_DownscalingLocal__4__4__4__a6f5bacb4f   has status   PENDING
DEBUG: Checking if CopyVolumeLocal(tmp_folder=tmp_zebrafish, max_jobs=1, config_dir=tmp_zebrafish\configs, input_path=C:\Users\meechan\Documents\test_images\zebrafish\0B51F8B46C_8bit_lynEGFP.tif, input_key=, output_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, output_key=setup0/timepoint0/s0, prefix=initial_scale, dtype=None, fit_to_roi=False, effective_scale_factor=[], dependency=DummyTask) is complete
INFO: Informed scheduler that task   DownscalingLocal_tmp_zebrafish_co_CopyVolumeLocal__2__2__2__7320a9867f   has status   PENDING
INFO: Informed scheduler that task   CopyVolumeLocal_tmp_zebrafish_co_DummyTask_None_7d7bb1ada8   has status   DONE
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 5
INFO: [pid 8728] Worker Worker(salt=086020834, workers=1, host=pc-schwab19, username=meechan, pid=8728) running   DownscalingLocal(tmp_folder=tmp_zebrafish, max_jobs=1, config_dir=tmp_zebrafish\configs, input_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, input_key=setup0/timepoint0/s0, output_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, output_key=setup0/timepoint0/s1, scale_factor=(2, 2, 2), scale_prefix=s1, halo=[2, 2, 2], effective_scale_factor=[2, 2, 2], dependency=CopyVolumeLocal)
ERROR: [pid 8728] Worker Worker(salt=086020834, workers=1, host=pc-schwab19, username=meechan, pid=8728) failed    DownscalingLocal(tmp_folder=tmp_zebrafish, max_jobs=1, config_dir=tmp_zebrafish\configs, input_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, input_key=setup0/timepoint0/s0, output_path=C:\Users\meechan\Documents\temp\minimal_mobie_project\data\zebrafish\images\local\membrane.n5, output_key=setup0/timepoint0/s1, scale_factor=(2, 2, 2), scale_prefix=s1, halo=[2, 2, 2], effective_scale_factor=[2, 2, 2], dependency=CopyVolumeLocal)
Traceback (most recent call last):
  File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\luigi\worker.py", line 191, in run
    new_deps = self._run_get_new_deps()
  File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\luigi\worker.py", line 133, in _run_get_new_deps
    task_gen = self.task.run()
  File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\cluster_tools\cluster_tasks.py", line 95, in run
    raise e
  File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\cluster_tools\cluster_tasks.py", line 81, in run
    self.run_impl()
  File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\cluster_tools\downscaling\downscaling.py", line 85, in run_impl
    with vu.file_reader(self.input_path, 'r') as f:
  File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\cluster_tools\utils\volume_utils.py", line 22, in file_reader
    return elf.io.open_file(path, mode=mode)
 File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\elf\io\files.py", line 39, in open_file
    return constructor(path, mode=mode)
  File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\z5py\file.py", line 85, in __init__
    raise OSError(errno.EROFS, os.strerror(errno.EROFS), handle.path())
OSError: [Errno 30] Read-only file system: 'C:\\Users\\meechan\\Documents\\temp\\minimal_mobie_project\\data\\zebrafish\\images\\local\\membrane.n5'
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   DownscalingLocal_tmp_zebrafish_co_CopyVolumeLocal__2__2__2__7320a9867f   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 5 pending tasks possibly being run by other workers
DEBUG: There are 5 pending tasks unique to this worker
DEBUG: There are 5 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=086020834, workers=1, host=pc-schwab19, username=meechan, pid=8728) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====
Scheduled 6 tasks of which:
* 1 complete ones were encountered:
    - 1 CopyVolumeLocal(...)
* 1 failed:
    - 1 DownscalingLocal(...)
* 4 were left pending, among these:
    * 4 had failed dependencies:
        - 2 DownscalingLocal(...)
        - 1 DownscalingWorkflow(...)
        - 1 WriteDownscalingMetadata(...)
This progress looks :( because there were failed tasks
===== Luigi Execution Summary =====
Traceback (most recent call last):
  File "<input>", line 12, in <module>
  File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\mobie\initialization.py", line 75, in initialize_dataset
    unit=unit)
  File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\mobie\import_data\raw.py", line 29, in import_raw_volume
    library='skimage', unit=unit)
  File "C:\Users\meechan\Anaconda3\envs\mobie\lib\site-packages\mobie\import_data\util.py", line 86, in downscale
    raise RuntimeError("Downscaling failed")
RuntimeError: Downscaling failed

[K-Meech]

I didn't copy the error properly - updated the comment above

[constantinpape]

Thanks for checking. I think that's a problem with windows file-paths...
I will try to find a windows machine to debug myself.

[constantinpape]

TODO add windows tests which use windows style file paths to see if this can be reproduced in CI

[CamachoDejay]

Hi,

Any update on this front. I am also getting the error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
c:\Users\xcamra\Documents\Jupyter-Notebooks\mobie-playground\minimal_example.ipynb Cell 2' in <cell line: 53>()
     [49](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=48) chunks = (1, 512, 512)
     [50](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=49) scale_factors = 4 * [[1, 2, 2]]
---> [53](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=52) mobie.add_image(
     [54](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=53)     input_path=input_file, 
     [55](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=54)     input_key='',  # the input is a single tif image, so we leave input_key blank
     [56](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=55)     root=mobie_project_folder,
     [57](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=56)     dataset_name=dataset_name,
     [58](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=57)     image_name=raw_name,
     [59](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=58)     menu_name=menu_name,
     [60](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=59)     resolution=resolution,
     [61](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=60)     chunks=chunks,
     [62](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=61)     scale_factors=scale_factors,
     [63](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=62)     is_default_dataset=True,  # mark this dataset as the default dataset that will be loaded by mobie
     [64](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=63)     target=target,
     [65](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=64)     max_jobs=max_jobs,
     [66](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=65)     unit=unit
     [67](vscode-notebook-cell:/c%3A/Users/xcamra/Documents/Jupyter-Notebooks/mobie-playground/minimal_example.ipynb#ch0000001?line=66) )

File c:\Users\xcamra\anaconda3\envs\mobie-env\lib\site-packages\mobie\image_data.py:227, in add_image(input_path, input_key, root, dataset_name, image_name, resolution, scale_factors, chunks, file_format, menu_name, tmp_folder, target, max_jobs, view, transformation, unit, is_default_dataset, description, move_only, int_to_uint)
    224         shutil.move(os.path.splitext(input_path)[0]+".xml", image_metadata_path)
    226 else:
--> 227     import_image_data(input_path, input_key, data_path,
...
    100 ret = luigi.build([t], local_scheduler=True)
    101 if not ret:
--> 102     raise RuntimeError("Downscaling failed")

RuntimeError: Downscaling failed

I am running this on Windows, python 3.9.13, and installed dependencies via “$ conda install -c conda-forge mobie_utils”
My jupyter-notebook looks something like minimal_example.txt

[constantinpape]

In fact I found another issue with running things on windows: constantinpape/cluster_tools#38
Interesting this means it should have never worked, so I am not quite sure how windows CI is passing. Maybe it's using some unix shell that has tail...
Anyway, I will update this here so that the native python tail implementation is used and then we can see if the problem is fixed or if there are more things to do.

[CamachoDejay]

I just wanted to let you know that after your latest fix propagated to anaconda forge and I updated the package I can now create the project. So from my end this is solved.

Thanks a lot!
Rafa

[constantinpape]

This is fixed now, thanks @CamachoDejay for testing!