Add DDIMVariantScheduler (torch-only DDIM variant, configurable prediction_type)#1
Add DDIMVariantScheduler (torch-only DDIM variant, configurable prediction_type)#1srlynch1 wants to merge 2 commits into
Conversation
…e prediction_type Copy-adapted from scheduling_ddim.py (no new abstraction, per the single-file scheduler convention): own DDIMVariantSchedulerOutput(BaseOutput), @register_to_config, 4-line dual-__init__ registration (alphabetical slot), generated dummy stub, and a SchedulerCommonTest test covering all three prediction_type values. ruff/check_copies/ check_dummies clean.
There was a problem hiding this comment.
Code Review
This pull request introduces DDIMVariantScheduler, a minimal, torch-only DDIM variant scheduler with a configurable prediction type, along with its documentation and a comprehensive test suite. The review feedback highlights critical issues regarding PyTorch tensor compatibility, specifically potential runtime errors when evaluating tensor conditionals in Python expressions (e.g., prev_timestep >= 0) and device mismatch errors when indexing CPU-bound cumulative product tensors with GPU tensors. The reviewer suggested resolving these by extracting scalar values using .item() and explicitly moving the scheduler's internal tensors to the active device.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| alpha_prod_t = self.alphas_cumprod[timestep] | ||
| alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep >= 0 else self.final_alpha_cumprod | ||
| beta_prod_t = 1 - alpha_prod_t | ||
| beta_prod_t_prev = 1 - alpha_prod_t_prev | ||
|
|
||
| variance = (beta_prod_t_prev / beta_prod_t) * (1 - alpha_prod_t / alpha_prod_t_prev) |
There was a problem hiding this comment.
When timestep or prev_timestep is passed as a PyTorch tensor (especially on a GPU device), evaluating prev_timestep >= 0 in a Python conditional expression will raise a RuntimeError because Python cannot implicitly convert a non-CPU tensor to a boolean.
To make this method robust and prevent runtime crashes when using tensor timesteps, we should extract the scalar value of prev_timestep using .item() if it is a tensor.
| alpha_prod_t = self.alphas_cumprod[timestep] | |
| alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep >= 0 else self.final_alpha_cumprod | |
| beta_prod_t = 1 - alpha_prod_t | |
| beta_prod_t_prev = 1 - alpha_prod_t_prev | |
| variance = (beta_prod_t_prev / beta_prod_t) * (1 - alpha_prod_t / alpha_prod_t_prev) | |
| prev_timestep_val = prev_timestep.item() if isinstance(prev_timestep, torch.Tensor) else prev_timestep | |
| alpha_prod_t = self.alphas_cumprod[timestep] | |
| alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep_val >= 0 else self.final_alpha_cumprod | |
| beta_prod_t = 1 - alpha_prod_t | |
| beta_prod_t_prev = 1 - alpha_prod_t_prev | |
| variance = (beta_prod_t_prev / beta_prod_t) * (1 - alpha_prod_t / alpha_prod_t_prev) |
| if self.num_inference_steps is None: | ||
| raise ValueError( | ||
| "Number of inference steps is 'None', you need to run 'set_timesteps' after creating the scheduler" | ||
| ) |
There was a problem hiding this comment.
If self.alphas_cumprod and self.final_alpha_cumprod remain on the CPU while timestep is a GPU tensor, indexing self.alphas_cumprod[timestep] will raise a device mismatch error.
To ensure correctness and avoid implicit device copies or runtime errors, we should explicitly move self.alphas_cumprod and self.final_alpha_cumprod to the device of model_output at the beginning of the step method, similar to how it is handled in add_noise and get_velocity.
| if self.num_inference_steps is None: | |
| raise ValueError( | |
| "Number of inference steps is 'None', you need to run 'set_timesteps' after creating the scheduler" | |
| ) | |
| if self.num_inference_steps is None: | |
| raise ValueError( | |
| "Number of inference steps is 'None', you need to run 'set_timesteps' after creating the scheduler" | |
| ) | |
| self.alphas_cumprod = self.alphas_cumprod.to(device=model_output.device) | |
| self.final_alpha_cumprod = self.final_alpha_cumprod.to(device=model_output.device) |
| # 1. get previous step value (=t-1) | ||
| prev_timestep = timestep - self.config.num_train_timesteps // self.num_inference_steps | ||
|
|
||
| # 2. compute alphas, betas | ||
| alpha_prod_t = self.alphas_cumprod[timestep] | ||
| alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep >= 0 else self.final_alpha_cumprod |
There was a problem hiding this comment.
When timestep is a PyTorch tensor (especially on a GPU device), prev_timestep will also be a tensor on the same device. Evaluating prev_timestep >= 0 in a Python conditional expression will raise a RuntimeError because Python cannot implicitly convert a non-CPU tensor to a boolean.
To make this method robust and prevent runtime crashes when using tensor timesteps, we should extract the scalar value of prev_timestep using .item() if it is a tensor.
| # 1. get previous step value (=t-1) | |
| prev_timestep = timestep - self.config.num_train_timesteps // self.num_inference_steps | |
| # 2. compute alphas, betas | |
| alpha_prod_t = self.alphas_cumprod[timestep] | |
| alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep >= 0 else self.final_alpha_cumprod | |
| # 1. get previous step value (=t-1) | |
| prev_timestep = timestep - self.config.num_train_timesteps // self.num_inference_steps | |
| prev_timestep_val = prev_timestep.item() if isinstance(prev_timestep, torch.Tensor) else prev_timestep | |
| # 2. compute alphas, betas | |
| alpha_prod_t = self.alphas_cumprod[timestep] | |
| alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep_val >= 0 else self.final_alpha_cumprod |
The ideate->brainstorm->plan->implement->test-coverage->review->ship stage docs that produced this scheduler (round-3 of the pack's e2e eval loop). Provenance for the feature in this PR; not library runtime content.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 3667c0e. Configure here.
What
Adds
DDIMVariantScheduler— a minimal, self-contained torch-only DDIM-variant scheduler with a configurableprediction_type(epsilon/sample/v_prediction).How
Copy-adapted from
src/diffusers/schedulers/scheduling_ddim.py(no new abstraction, per the single-file scheduler convention):DDIMVariantScheduler(SchedulerMixin, ConfigMixin)with@register_to_configDDIMVariantSchedulerOutput(BaseOutput);step()returns it (never a bare tuple)# Copied from diffusers.schedulers.scheduling_ddpm.*blocks carried over byte-identical (the output-dataclass one rewrittenwith DDPM->DDIMVariant)__init__registration in alphabetical slot (scheduling_ddim_variantbetween_paralleland_ddpm); generated dummy stub indummy_pt_objects.pytests/schedulers/test_scheduler_ddim_variant.pysubclassingSchedulerCommonTest, covering all threeprediction_typevalues_toctree.ymlentryGates
Clean locally:
ruff check,ruff format --check,python utils/check_copies.py(exit 0),python utils/check_dummies.py(exit 0),check_doc_toc.py.make qualitypart 3 (doc-builder) and therun_fast_testspytest matrix are CI-only on this host — CI is the final arbiter.Provenance
Produced end-to-end by the diffusers Self-Improving Pack's SDLC chain (
ideate → brainstorm → plan → implement → test-coverage → review → ship) as a real eval, then iterated over 3 eval→improve→eval rounds.Note
Low Risk
Purely additive public API with no auth or data-path changes; main maintainer risk is long-term sync with identical DDIM behavior unless a class-level
# Copied fromlink is added later.Overview
Adds
DDIMVariantScheduleras a new public, torch-only DDIM-family scheduler with configurableprediction_type(epsilon,sample,v_prediction), implemented by copy-adaptingscheduling_ddim.pyintoscheduling_ddim_variant.py(single-file convention; no changes toDDIMScheduler).Library wiring: exports in
src/diffusers/__init__.pyandsrc/diffusers/schedulers/__init__.py, adummy_pt_objects.pystub, API pageddim_variant.md, and a_toctree.ymlentry.Tests: new
test_scheduler_ddim_variant.pyonSchedulerCommonTest, mirroring DDIM tests with broaderprediction_typecoverage (all three branches plus invalid-typeValueError).Docs: seven new
docs/e2e-run/*artifacts (ideation through ship) documenting the SDLC/eval chain; not runtime code.Reviewed by Cursor Bugbot for commit 3667c0e. Bugbot is set up for automated code reviews on this repo. Configure here.