Skip to content

Multi diffusion#1462

Merged
CharlelieLrt merged 26 commits intoNVIDIA:mainfrom
CharlelieLrt:multi-diffusion
Mar 11, 2026
Merged

Multi diffusion#1462
CharlelieLrt merged 26 commits intoNVIDIA:mainfrom
CharlelieLrt:multi-diffusion

Conversation

@CharlelieLrt
Copy link
Copy Markdown
Collaborator

PhysicsNeMo Pull Request

Description

Checklist

Dependencies

Review Process

All PRs are reviewed by the PhysicsNeMo team before merging.

Depending on which files are changed, GitHub may automatically assign a maintainer for review.

We are also testing AI-based code review tools (e.g., Greptile), which may add automated comments with a confidence score.
This score reflects the AI’s assessment of merge readiness and is not a qualitative judgment of your work, nor is
it an indication that the PR will be accepted / rejected.

AI-generated feedback should be reviewed critically for usefulness.
You are not required to respond to every AI comment, but they are intended to help both authors and reviewers.
Please react to Greptile comments with 👍 or 👎 to provide feedback on their accuracy.

jleinonen and others added 18 commits February 19, 2026 13:37
* Bug fixes for ShardTensor+SongUNet

* Handle dtensor spec in sharded view

* Fix SongUNet with ShardTensor when using zero embedding

* Use buffer for zero embed

---------

Co-authored-by: Peter Harrington <48932392+pzharrington@users.noreply.github.com>
Co-authored-by: Peter Harrington <pharrington@nvidia.com>
* comment out e2grid and makani installs

* fix dtype

* update sfno test

* update version

* some fixes for healpix tests, update to nc install, fix test dir path mismatch

* revert changes to healpix code

* don't change the default path

* fix graphcast doctest, update doctest command to ignore onnx module because of conflict with onnx.utils

* skip pytest for debugging, use floating point comparison for gumbel softmax

* make the test more robust

* bring back pytests
…DIA#1433)

* Bugfix with num_steps parameters in CorrDiff generate.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Update examples/weather/corrdiff/conf/base/generation/sampler/stochastic.yaml

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
NVIDIA#1430)

* DSMLoss implementation + minor bugfixes in noise_schedulers.py and preconditioners.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Bugfix for missing abstract class in noise_schedulers.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Grammar improvements

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Added 'reduction' argument to MSEDSMLoss + added new WeightedMSEDSMLoss

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

---------

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
…losses

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
…losses

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
@CharlelieLrt CharlelieLrt self-assigned this Feb 27, 2026
@CharlelieLrt CharlelieLrt added the 5 - Merge After Dependencies Depends on another PR: do not merge out of order label Feb 27, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Feb 27, 2026

Greptile Summary

This PR implements the multi-diffusion functionality for patch-based diffusion model training and inference, replacing the previous placeholder implementation. The changes introduce comprehensive support for decomposing 2D images into smaller patches, processing them through diffusion models, and fusing them back together.

Key changes:

  • Implemented MultiDiffusionModel2D wrapper that handles patching strategies (random for training, grid for inference), condition preprocessing, and optional positional embeddings
  • Added two loss classes (MultiDiffusionMSEDSMLoss and MultiDiffusionWeightedMSEDSMLoss) for patch-based denoising score matching training
  • Refactored patching.py from a placeholder warning to a full implementation with BasePatching2D, RandomPatching2D, and GridPatching2D classes
  • Added comprehensive test suite with reference data for non-regression testing
  • Improved test organization by moving shared fixtures to conftest.py
  • Updated import linter configuration to include the new multi-diffusion module

Code quality:

  • Excellent documentation with detailed docstrings following NumPy style
  • Comprehensive examples in docstrings demonstrating various use cases
  • Proper use of jaxtyping for type hints
  • Good test coverage with reference data for validation

Minor issues found:

  • Two instances of typo: "postional" should be "positional" in comment lines

Important Files Changed

Filename Overview
physicsnemo/diffusion/multi_diffusion/losses.py New file implementing multi-diffusion denoising score matching losses with comprehensive docstrings and examples. Contains minor typo in comments.
physicsnemo/diffusion/multi_diffusion/models.py New file implementing MultiDiffusionModel2D wrapper for patch-based diffusion with excellent documentation and examples.
physicsnemo/diffusion/multi_diffusion/patching.py Refactored from placeholder to full implementation of 2D image patching utilities (random and grid-based) with comprehensive documentation.
physicsnemo/diffusion/multi_diffusion/init.py Exports new multi-diffusion components: loss classes, model wrapper, and patching utilities.
test/diffusion/test_patching.py New comprehensive test suite for patching utilities with reference data for non-regression testing.

Last reviewed commit: 2e6b052

Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

30 files reviewed, 2 comments

Edit Code Review Agent Settings | Greptile

Comment thread physicsnemo/diffusion/multi_diffusion/losses.py Outdated
Comment thread physicsnemo/diffusion/multi_diffusion/losses.py Outdated
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
@CharlelieLrt CharlelieLrt removed 5 - Merge After Dependencies Depends on another PR: do not merge out of order labels Mar 6, 2026
Comment thread physicsnemo/diffusion/multi_diffusion/models.py
Comment thread physicsnemo/diffusion/multi_diffusion/losses.py Outdated
Copy link
Copy Markdown
Collaborator

@pzharrington pzharrington left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, just minor comments!

@CharlelieLrt
Copy link
Copy Markdown
Collaborator Author

/blossom-ci

@CharlelieLrt CharlelieLrt enabled auto-merge March 10, 2026 21:54
Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
@CharlelieLrt
Copy link
Copy Markdown
Collaborator Author

/blossom-ci

@CharlelieLrt CharlelieLrt added this pull request to the merge queue Mar 11, 2026
Merged via the queue into NVIDIA:main with commit dfa2e2a Mar 11, 2026
4 checks passed
nbren12 pushed a commit to nbren12/modulus that referenced this pull request Mar 24, 2026
* Fix SongUNet with ShardTensor when using zero embedding (NVIDIA#1432)

* Bug fixes for ShardTensor+SongUNet

* Handle dtensor spec in sharded view

* Fix SongUNet with ShardTensor when using zero embedding

* Use buffer for zero embed

---------

Co-authored-by: Peter Harrington <48932392+pzharrington@users.noreply.github.com>
Co-authored-by: Peter Harrington <pharrington@nvidia.com>

* Few fixes (NVIDIA#1434)

* comment out e2grid and makani installs

* fix dtype

* update sfno test

* update version

* some fixes for healpix tests, update to nc install, fix test dir path mismatch

* revert changes to healpix code

* don't change the default path

* fix graphcast doctest, update doctest command to ignore onnx module because of conflict with onnx.utils

* skip pytest for debugging, use floating point comparison for gumbel softmax

* make the test more robust

* bring back pytests

* Bugfix with missing num_steps parameters in CorrDiff generate.py (NVIDIA#1433)

* Bugfix with num_steps parameters in CorrDiff generate.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Update examples/weather/corrdiff/conf/base/generation/sampler/stochastic.yaml

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* DSMLoss implementation + minor bugfixes in noise_schedulers.py and pr… (NVIDIA#1430)

* DSMLoss implementation + minor bugfixes in noise_schedulers.py and preconditioners.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Bugfix for missing abstract class in noise_schedulers.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Grammar improvements

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Added 'reduction' argument to MSEDSMLoss + added new WeightedMSEDSMLoss

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

---------

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Initial version of multi-diffusion model wrapper and multi-diffusion losses

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Improvements to multi-diffusion model wrapper

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Refactored patching.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* New tests for patching utilities in patching.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Refactored patching.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Fixed global_index reference in patching.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Added docstring examples + recompile tests for patching.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Initial version of multi-diffusion model wrapper and multi-diffusion losses

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Improvements to multi-diffusion model wrapper

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Completed implementation of multi-diffusion model wrapper

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Completed implementation of multi-diffusion losses

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Added tests for multi_diffusion/models.py

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Added tests for multi-diffusion losses

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Added option + note about automated redrawing of patch indices

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Fixes typos

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

* Disable non-regression for tests on cuda

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>

---------

Signed-off-by: Charlelie Laurent <claurent@nvidia.com>
Co-authored-by: Jussi Leinonen <jleinonen@nvidia.com>
Co-authored-by: Peter Harrington <48932392+pzharrington@users.noreply.github.com>
Co-authored-by: Peter Harrington <pharrington@nvidia.com>
Co-authored-by: Kaustubh Tangsali <71059996+ktangsali@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants