Question about long-time training

[leeruibin]

When I followed the training instructions to reproduce the results, I encountered some issues during replication. For example, as described in the paper, I fine-tuned the model for 3K steps, but the fine-tuned model’s performance was noticeably worse than that of the provided checkpoint.

This is the result from the reproduced 3K ckpt.

A.cinematic.scene.from.a.classic.western.movie.featuring.a.rugged.man.riding.a.powerful.horse.throu-0.7.mp4

The motion can become very slow.

And when I extended the training to around 15K steps, the model began to show strange camera movements and repetitive content.

A.cinematic.scene.from.a.classic.western.movie.featuring.a.rugged.man.riding.a.powerful.horse.throu-0.10.mp4

And this is the inference result from the provided ckpt.

A.cinematic.scene.from.a.classic.western.movie.featuring.a.rugged.man.riding.a.powerful.horse.throu-0.9.mp4

The motion seems to be normal.

Do you have any ideas about why this happen?

[mingzhe11]

How about your inference speed?

[Fangzhou-Ai]

The inference speed is the same as claimed by the author, but the quality of the video trained following the README is much more degregated compared with the ckpt offered by the author.

[bliblioscar]

I have the same question with you! The motion become very slow in my reproduction. @leeruibin

[SARIHUST]

Hi, I also ran into similar issues when reproducing the results.

In my case, the degradation appears even earlier — at 3K iterations, it already appears. The model begins to show noticeable camera drift / unintended camera movements, and the main subject sometimes disappears entirely. Then, in order to “follow” the prompt, the subject reappears suddenly and continues the motion, but the overall temporal consistency becomes quite strange.
This behavior looks very similar to what others reported at around 15K steps, just happening earlier for me.

Would love to know if anyone has insights into why this happens or if there are recommended hyperparameters/configs to stabilize the training.

A.close-up.3D.animated.scene.of.a.short.fluffy.monster.kneeling.beside.a.melting.red.candle.The.mo-0.mp4

A.close-up.shot.of.a.Victoria.crowned.pigeon.in.a.naturalistic.wildlife.photography.style.showcasin-0.mp4

A.stylish.woman.strolls.down.a.bustling.Tokyo.street.the.warm.glow.of.neon.lights.and.animated.city-0.mp4

[my4f175]

I'm encountering the same problem. The camera keeps panning left. Is there any update on this issue?

[ealicesora]

Hi, I'm wondering if it is caused by the ode-init ckpt released by the authors? Do people encounter the same problem use the ode-init ckpt provided by the author, or trained from scratch?

[zhuhz22]

Hi, I'm wondering if it is caused by the ode-init ckpt released by the authors? Do people encounter the same problem use the ode-init ckpt provided by the author, or trained from scratch?

Please refer to our Causal Forcing that may help: we identify a mathematical fallacy in the ODE initialization stage of Self Forcing and CausVid, and propose an efficient, stable, and high-quality ODE initialization method that significantly outperforms them in both visual quality and motion dynamics, while maintaining the same training budget and inference efficiency.

• Code (full-stack open source): https://github.com/thu-ml/Causal-Forcing
• Page: https://thu-ml.github.io/CausalForcing.github.io/

[ealicesora]

Hi, I'm wondering if it is caused by the ode-init ckpt released by the authors? Do people encounter the same problem use the ode-init ckpt provided by the author, or trained from scratch?

Please refer to our Causal Forcing that may help: we identify a mathematical fallacy in the ODE initialization stage of Self Forcing and CausVid, and propose an efficient, stable, and high-quality ODE initialization method that significantly outperforms them in both visual quality and motion dynamics, while maintaining the same training budget and inference efficiency.

• Code (full-stack open source): https://github.com/thu-ml/Causal-Forcing
• Page: https://thu-ml.github.io/CausalForcing.github.io/

@zhuhz22 Thanks for your reply. In fact, I read your work causal forcing yesterday, and that's why I am asking the question here.(BTW, thanks for the great work.) I believe it is the ode-init.pt released by the author caused the camera drift issue mentioned above. However, the final ckpt released by the author may use another way for the ode-init(adding noise in the mse), which result in better performance. I plan to perform an experiment on it, and maybe come back when I get some results.

[zhuhz22]

Hi @ealicesora ,
We reproduced Rolling Forcing using Causal Forcing’s causal ODE initialization, and the issue seems to be alleviated: with roughly 3K training steps, we can already achieve strong results without the abnormal camera panning:

A.cinematic.scene.from.a.classic.western.movie.featuring.a.rugged.man.riding.a.powerful.horse.throu-0.mp4

A.dramatic.post-apocalyptic.scene.in.the.style.of.a.horror.film.featuring.a.skeleton.wearing.a.colo-0.mp4

Thus, we suggest trying to replace the ODE model in Self Forcing with ours.

We change the config as follows to keep it aligned with our causal pretraining:

denoising_step_list:
  - 1000
  - 750
  - 500
  - 250

[ealicesora]

Hi @zhuhz22 , thank you so much for running the experiments and sharing the results. It seems really work! Really appreciate your help!

[ealicesora]

Hi! @zhuhz22 I am trying to replicate the results with rolling forcing and your correct init ckpt (checkpoints/framewise/causal_ode.pt) with 2k iters, but I am still suffering the abnormal camera panning, while the dynamics looks much better than before. (I am using the new denoising_step_list you provided before). Do you make any other adjustment?
Below is the results produced by my 2k training ckpt:

A.cinematic.scene.from.a.classic.western.movie.featuring.a.rugged.man.riding.a.powerful.horse.throu-0.mp4

[zhuhz22]

I used the chunk-wise model here. Or how about training for 1K or 3K? Our reproduced Rolling Forcing has no abnormal camera panning.

[ealicesora]

Hi @zhuhz22 , thanks for your assistance.

Since I configured the trainer to save a checkpoint every 2k iterations, I’ll rerun the training process and try to reproduce the issue. I also checked the YAML you shared,(https://github.com/thu-ml/Causal-Forcing/blob/main/long_video/configs/rolling_forcing_dmd.yaml) it looks almost identical to my configuration.

One key difference is the hardware setup: I’m training with 8×H200 only. With this setup, my global batch size is 8 (i.e., batch size per step = 8 across all GPUs), which is quite different from the paper’s setting using 64 GPUs.

Could you please share how many GPUs you used to train Rolling Forcing with the chunk-wise checkpointing setting?

Thanks again!

[zhuhz22]

Hi @zhuhz22 , thanks for your assistance.

Since I configured the trainer to save a checkpoint every 2k iterations, I’ll rerun the training process and try to reproduce the issue. I also checked the YAML you shared,(https://github.com/thu-ml/Causal-Forcing/blob/main/long_video/configs/rolling_forcing_dmd.yaml) it looks almost identical to my configuration.

One key difference is the hardware setup: I’m training with 8×H200 only. With this setup, my global batch size is 8 (i.e., batch size per step = 8 across all GPUs), which is quite different from the paper’s setting using 64 GPUs.

Could you please share how many GPUs you used to train Rolling Forcing with the chunk-wise checkpointing setting?

Thanks again!

I use 64 A800 with bs=64.