Skip to content

Add DDIMVariantScheduler (torch-only DDIM variant, configurable prediction_type)#1

Open
srlynch1 wants to merge 2 commits into
mainfrom
eval/e2e-ddim-variant-r3
Open

Add DDIMVariantScheduler (torch-only DDIM variant, configurable prediction_type)#1
srlynch1 wants to merge 2 commits into
mainfrom
eval/e2e-ddim-variant-r3

Conversation

@srlynch1

@srlynch1 srlynch1 commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

What

Adds DDIMVariantScheduler — a minimal, self-contained torch-only DDIM-variant scheduler with a configurable prediction_type (epsilon / sample / v_prediction).

How

Copy-adapted from src/diffusers/schedulers/scheduling_ddim.py (no new abstraction, per the single-file scheduler convention):

  • DDIMVariantScheduler(SchedulerMixin, ConfigMixin) with @register_to_config
  • own DDIMVariantSchedulerOutput(BaseOutput); step() returns it (never a bare tuple)
  • the five # Copied from diffusers.schedulers.scheduling_ddpm.* blocks carried over byte-identical (the output-dataclass one rewritten with DDPM->DDIMVariant)
  • 4-line dual-__init__ registration in alphabetical slot (scheduling_ddim_variant between _parallel and _ddpm); generated dummy stub in dummy_pt_objects.py
  • tests/schedulers/test_scheduler_ddim_variant.py subclassing SchedulerCommonTest, covering all three prediction_type values
  • API docs page + _toctree.yml entry

Gates

Clean locally: ruff check, ruff format --check, python utils/check_copies.py (exit 0), python utils/check_dummies.py (exit 0), check_doc_toc.py. make quality part 3 (doc-builder) and the run_fast_tests pytest matrix are CI-only on this host — CI is the final arbiter.

Provenance

Produced end-to-end by the diffusers Self-Improving Pack's SDLC chain (ideate → brainstorm → plan → implement → test-coverage → review → ship) as a real eval, then iterated over 3 eval→improve→eval rounds.


Note

Low Risk
Purely additive public API with no auth or data-path changes; main maintainer risk is long-term sync with identical DDIM behavior unless a class-level # Copied from link is added later.

Overview
Adds DDIMVariantScheduler as a new public, torch-only DDIM-family scheduler with configurable prediction_type (epsilon, sample, v_prediction), implemented by copy-adapting scheduling_ddim.py into scheduling_ddim_variant.py (single-file convention; no changes to DDIMScheduler).

Library wiring: exports in src/diffusers/__init__.py and src/diffusers/schedulers/__init__.py, a dummy_pt_objects.py stub, API page ddim_variant.md, and a _toctree.yml entry.

Tests: new test_scheduler_ddim_variant.py on SchedulerCommonTest, mirroring DDIM tests with broader prediction_type coverage (all three branches plus invalid-type ValueError).

Docs: seven new docs/e2e-run/* artifacts (ideation through ship) documenting the SDLC/eval chain; not runtime code.

Reviewed by Cursor Bugbot for commit 3667c0e. Bugbot is set up for automated code reviews on this repo. Configure here.

…e prediction_type

Copy-adapted from scheduling_ddim.py (no new abstraction, per the single-file
scheduler convention): own DDIMVariantSchedulerOutput(BaseOutput), @register_to_config,
4-line dual-__init__ registration (alphabetical slot), generated dummy stub, and a
SchedulerCommonTest test covering all three prediction_type values. ruff/check_copies/
check_dummies clean.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces DDIMVariantScheduler, a minimal, torch-only DDIM variant scheduler with a configurable prediction type, along with its documentation and a comprehensive test suite. The review feedback highlights critical issues regarding PyTorch tensor compatibility, specifically potential runtime errors when evaluating tensor conditionals in Python expressions (e.g., prev_timestep >= 0) and device mismatch errors when indexing CPU-bound cumulative product tensors with GPU tensors. The reviewer suggested resolving these by extracting scalar values using .item() and explicitly moving the scheduler's internal tensors to the active device.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +281 to +286
alpha_prod_t = self.alphas_cumprod[timestep]
alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep >= 0 else self.final_alpha_cumprod
beta_prod_t = 1 - alpha_prod_t
beta_prod_t_prev = 1 - alpha_prod_t_prev

variance = (beta_prod_t_prev / beta_prod_t) * (1 - alpha_prod_t / alpha_prod_t_prev)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

When timestep or prev_timestep is passed as a PyTorch tensor (especially on a GPU device), evaluating prev_timestep >= 0 in a Python conditional expression will raise a RuntimeError because Python cannot implicitly convert a non-CPU tensor to a boolean.

To make this method robust and prevent runtime crashes when using tensor timesteps, we should extract the scalar value of prev_timestep using .item() if it is a tensor.

Suggested change
alpha_prod_t = self.alphas_cumprod[timestep]
alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep >= 0 else self.final_alpha_cumprod
beta_prod_t = 1 - alpha_prod_t
beta_prod_t_prev = 1 - alpha_prod_t_prev
variance = (beta_prod_t_prev / beta_prod_t) * (1 - alpha_prod_t / alpha_prod_t_prev)
prev_timestep_val = prev_timestep.item() if isinstance(prev_timestep, torch.Tensor) else prev_timestep
alpha_prod_t = self.alphas_cumprod[timestep]
alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep_val >= 0 else self.final_alpha_cumprod
beta_prod_t = 1 - alpha_prod_t
beta_prod_t_prev = 1 - alpha_prod_t_prev
variance = (beta_prod_t_prev / beta_prod_t) * (1 - alpha_prod_t / alpha_prod_t_prev)

Comment on lines +428 to +431
if self.num_inference_steps is None:
raise ValueError(
"Number of inference steps is 'None', you need to run 'set_timesteps' after creating the scheduler"
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If self.alphas_cumprod and self.final_alpha_cumprod remain on the CPU while timestep is a GPU tensor, indexing self.alphas_cumprod[timestep] will raise a device mismatch error.

To ensure correctness and avoid implicit device copies or runtime errors, we should explicitly move self.alphas_cumprod and self.final_alpha_cumprod to the device of model_output at the beginning of the step method, similar to how it is handled in add_noise and get_velocity.

Suggested change
if self.num_inference_steps is None:
raise ValueError(
"Number of inference steps is 'None', you need to run 'set_timesteps' after creating the scheduler"
)
if self.num_inference_steps is None:
raise ValueError(
"Number of inference steps is 'None', you need to run 'set_timesteps' after creating the scheduler"
)
self.alphas_cumprod = self.alphas_cumprod.to(device=model_output.device)
self.final_alpha_cumprod = self.final_alpha_cumprod.to(device=model_output.device)

Comment on lines +444 to +449
# 1. get previous step value (=t-1)
prev_timestep = timestep - self.config.num_train_timesteps // self.num_inference_steps

# 2. compute alphas, betas
alpha_prod_t = self.alphas_cumprod[timestep]
alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep >= 0 else self.final_alpha_cumprod

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

When timestep is a PyTorch tensor (especially on a GPU device), prev_timestep will also be a tensor on the same device. Evaluating prev_timestep >= 0 in a Python conditional expression will raise a RuntimeError because Python cannot implicitly convert a non-CPU tensor to a boolean.

To make this method robust and prevent runtime crashes when using tensor timesteps, we should extract the scalar value of prev_timestep using .item() if it is a tensor.

Suggested change
# 1. get previous step value (=t-1)
prev_timestep = timestep - self.config.num_train_timesteps // self.num_inference_steps
# 2. compute alphas, betas
alpha_prod_t = self.alphas_cumprod[timestep]
alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep >= 0 else self.final_alpha_cumprod
# 1. get previous step value (=t-1)
prev_timestep = timestep - self.config.num_train_timesteps // self.num_inference_steps
prev_timestep_val = prev_timestep.item() if isinstance(prev_timestep, torch.Tensor) else prev_timestep
# 2. compute alphas, betas
alpha_prod_t = self.alphas_cumprod[timestep]
alpha_prod_t_prev = self.alphas_cumprod[prev_timestep] if prev_timestep_val >= 0 else self.final_alpha_cumprod

The ideate->brainstorm->plan->implement->test-coverage->review->ship stage docs
that produced this scheduler (round-3 of the pack's e2e eval loop). Provenance for
the feature in this PR; not library runtime content.
@srlynch1

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 3667c0e. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant