The current model configuration doesn't work well on the line dataset because the timestep is not large enough. #4

Activity

1. toannguyen1904 commented on Dec 27, 2023

   

   toannguyen1904

   on Dec 27, 2023

   More actions

   As the ablation study in readme.md shows, the current model configuration doesn't work well on the line dataset. The reason is the the timestep is not large enough such that Xt is not in standard gaussian distribution. If we set timestep as 250 instead of 50, the result is nice.

   Helpful!
2. sanjayBahadur commented on Oct 17, 2026

   

   sanjayBahadur

   on Oct 17, 2026

   More actions

   Adding a reproducible numerical diagnostic for the terminal-distribution mismatch discussed above. At commit 726b6bb, the CLI uses 50 timesteps and linear betas from 0.0001 to 0.02 (the scheduler class itself defaults to 1,000).

   For a = product(1 - beta_t), the last forward marginal is sqrt(a) * x_0 + sqrt(1-a) * epsilon:

   Timesteps	Terminal cumulative alpha	Remaining signal coefficient	Terminal coordinate variances
   50	0.6029515973	0.7764995797	(1.2010, 3.6128)
   250	0.0797025443	0.2823163905	(1.0266, 1.3454)

   The variance column uses the population distribution in line_dataset: independent uniforms on [-2, 2] and [-4, 4], with variances (4/3, 16/3). Hence Cov(x_T) = a * Cov(x_0) + (1-a) * I. Sampling starts from a standard Gaussian with variances (1, 1).

   This can be checked with Python's standard library alone:

   import math
   
   for steps in (50, 250):
       betas = [0.0001 + (0.02 - 0.0001) * i / (steps - 1)
                for i in range(steps)]
       a = math.prod(1 - b for b in betas)
       variances = tuple(a * v + (1 - a) for v in (4 / 3, 16 / 3))
       print(steps, a, math.sqrt(a), variances)

   These double-precision calculations support investigating the mismatch; they do not reproduce the training result or establish its sole cause. Increasing timesteps with fixed beta endpoints changes total forward corruption as well as sampling cost, and 250 steps still do not yield unit terminal covariance. Covariance agreement alone would also be insufficient to prove a Gaussian marginal.

   For a follow-up experiment, reporting these diagnostics alongside generated samples would distinguish changes to the training noise schedule from changes to inference cost. Reducing inference evaluations while holding trained weights fixed needs an appropriate sampler such as DDIM; skipping entries in the existing ancestral loop is not equivalent.