Skip to content

5385-enhance-mlflow-handler#5388

Merged
wyli merged 63 commits into
Project-MONAI:devfrom
binliunls:5385-enhance-mlflow-handler
Dec 2, 2022
Merged

5385-enhance-mlflow-handler#5388
wyli merged 63 commits into
Project-MONAI:devfrom
binliunls:5385-enhance-mlflow-handler

Conversation

@binliunls

@binliunls binliunls commented Oct 24, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Signed-off-by: binliu binliu@nvidia.com

Fixes #5385 .

Description

This PR is about to enhance the mlflow handler in monai to track more details in the experiment. Here are a few enhancements
that needs to be added through this PR.

  • API for users to add experiment/run name in MLFlow
  • API for users to log customized params for each run
  • Methods to log result images
  • Methods to log optimizer params
  • (optional) additional metric_names as a user argument to override the default engine.state.metrics to instruct MLFlow about metrics to log

After adding these enhancements, some tests listed below should be excuted.

  • Make sure this handler works in multi-gpu environment
  • Make sure this handler works in all existed bundles

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.
  • API for users to add experiment/run name in MLFlow
  • API for users to log customized params for each run
  • Methods to log result images
  • Methods to log optimizer params
  • Make sure this handler works in multi-gpu environment
  • Make sure this handler works in all existed bundles

@binliunls
add experiment/run name to mlflow handler
695bc74 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
add apis to mlflow handler
b88a089 
Signed-off-by: binliu <binliu@nvidia.com>
mingxin-zheng
mingxin-zheng reviewed Oct 26, 2022
View reviewed changes
Comment thread monai/handlers/mlflow_handler.py Outdated
mingxin-zheng
mingxin-zheng reviewed Oct 26, 2022
View reviewed changes
Comment thread monai/handlers/mlflow_handler.py Outdated
mingxin-zheng
mingxin-zheng reviewed Oct 26, 2022
View reviewed changes
Comment thread monai/handlers/mlflow_handler.py
binliunls and others added 4 commits October 27, 2022 07:08
mingxin-zheng
mingxin-zheng reviewed Oct 28, 2022
View reviewed changes
Comment thread monai/handlers/mlflow_handler.py Outdated
binliunls and others added 3 commits November 2, 2022 15:29
Nic-Ma
Nic-Ma reviewed Nov 3, 2022
View reviewed changes

@Nic-Ma Nic-Ma left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update, put some comments inline.
Need @wyli @ericspod @vfdev-5 for more design discussion.

Thanks in advance.

Comment thread monai/handlers/mlflow_handler.py Outdated
Comment thread monai/handlers/mlflow_handler.py Outdated
Comment thread monai/handlers/mlflow_handler.py
Comment thread monai/handlers/mlflow_handler.py Outdated
@Nic-Ma

Nic-Ma commented Nov 11, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Please ensure we can call these get_config_values() and get_optional_config_values() functions to track the system env info in MLFlow:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/config/deviceconfig.py#L48-L81

CC @wyli

Thanks.

binliunls and others added 12 commits November 15, 2022 14:34
@binliunls
merge dev branch
736053b 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
change mlflow run name according to input parameter.
34e3a8a 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
merge dev branch to current branch
8325cf3 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
format code
a92eb19 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
Merge branch 'Project-MONAI:dev' into 5385-enhance-mlflow-handler
765061b
@binliunls
fix codeformat problem
95227f7 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
Merge branch '5385-enhance-mlflow-handler' of https://github.com/binl…
6ee0dbf 
…iunls/MONAI into 5385-enhance-mlflow-handler
@binliunls
update the code comments and handler for validator
5d4475f 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
update some type hints and comments
e8760d7 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
update default mlflow tracking with bundle info and multi gpu info
7fe8269 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
fix the log default parameter bug
39f40b6 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
add disable for multi gpu in validator and evaluator
e3d2cb3 
Signed-off-by: binliu <binliu@nvidia.com>
Nic-Ma
Nic-Ma reviewed Dec 1, 2022
View reviewed changes
Comment thread monai/handlers/mlflow_handler.py Outdated
Nic-Ma
Nic-Ma reviewed Dec 1, 2022
View reviewed changes
Comment thread monai/handlers/mlflow_handler.py Outdated
binliunls and others added 7 commits December 1, 2022 07:35
@binliunls
change the default value of mlflow run name in untils
28aae20 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
change the not_rank0 name to is_not_rank0 in bundle/utils
c564df4 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
change some arguments' type in mlflow hander
1e95f12 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
delete a legacy method in mlflow handler
b7fcff1 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
remove the trainer check logic from the mlflow handler since users sh…
1c7437b 
…ould set iteration_log=False to avoid this

Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
remove the Trainer import, since it won't be used in mlflow handler
608ce24 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
Merge branch 'Project-MONAI:dev' into 5385-enhance-mlflow-handler
0961783
wyli
wyli reviewed Dec 1, 2022
View reviewed changes

@wyli wyli left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I put some minor comments inline

Comment thread monai/bundle/utils.py Outdated
Comment thread monai/bundle/utils.py Outdated
Comment thread monai/handlers/mlflow_handler.py Outdated
binliunls and others added 4 commits December 1, 2022 12:44
@binliunls
change default name to monai experiment and change the rank0 logic in…
ecab937 
… bundle/utils

Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
merge two functions into one
cb7ae68 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
format code style in mlflow handler
48507f3 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
Merge branch 'Project-MONAI:dev' into 5385-enhance-mlflow-handler
22036b7
wyli
wyli approved these changes Dec 1, 2022
View reviewed changes

@wyli wyli left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me, @Nic-Ma pls let me know if you still have comments otherwise I'll merge this later today

@wyli

wyli commented Dec 1, 2022

Copy link
Copy Markdown
Contributor

/build

Nic-Ma
Nic-Ma requested changes Dec 1, 2022
View reviewed changes
Comment thread monai/bundle/utils.py Outdated
Comment thread monai/handlers/mlflow_handler.py Outdated
Comment thread monai/handlers/mlflow_handler.py
@Nic-Ma

Nic-Ma commented Dec 1, 2022

Copy link
Copy Markdown
Contributor

Could you please help also add the experiment_name and run_name in this doc-string example:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/bundle/scripts.py#L549

Thanks.

@binliunls
change default run name in mlflow to None
a8fef29 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
update doc string of default value of optimizer_param_names
de48bee 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
udpate default mlflow setting docstring in bundle/script
56fbb05 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
fix code format issue in mlflow handler
850f98f 
Signed-off-by: binliu <binliu@nvidia.com>
Nic-Ma
Nic-Ma reviewed Dec 2, 2022
View reviewed changes
Comment thread monai/bundle/scripts.py
@binliunls
change the default_attr_name to default_tracking_params to avoid conf…
ebb804c 
…usion

Signed-off-by: binliu <binliu@nvidia.com>
@Nic-Ma Nic-Ma requested a review from wyli December 2, 2022 15:47
@Nic-Ma

Nic-Ma commented Dec 2, 2022

Copy link
Copy Markdown
Contributor

/build

@wyli wyli enabled auto-merge (squash) December 2, 2022 16:07
@wyli wyli merged commit 7b41e2e into Project-MONAI:dev Dec 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mingxin-zheng mingxin-zheng mingxin-zheng left review comments
@Nic-Ma Nic-Ma Nic-Ma approved these changes
@wyli wyli Awaiting requested review from wyli

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Enhance MLFlowHandler

4 participants

@binliunls @Nic-Ma @wyli @mingxin-zheng