📽️ VideoFrom3D 📽️
_{3D Scene Video Generation via Complementary Image and Video Diffusion Models}

Geonung Kim  Janghyeok Han  Sunghyun Cho 

POSTECH CG Lab.

SIGGRAPH-ASIA 2025 Conference

🔥 Update

• [2025.10.27] Training code for GGI Module is released.
• [2025.10.10] Preprocessing code is released.
• [2025.09.08] The repository is created.

⚙️ Preprocessing

The detailed preprocessing procedure, such as edge and flow extracation, is explained at VideoFrom3D/assets/readme.md. Please refer to this document.

⌨️ Training GGI

The detailed training procedure is explained at VideoFrom3D/training_ggi/readme.md. Please refer to this document.

⌨️ Inference

Install Environment

# create conda environment
conda create --name videofrom3d python=3.10

# install pytorch (use propor cuda version option)
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121

# install other packages
pip install -r requirements.txt
 

Sparse Appearance-guided Sampling (SAG)

Distribution Alignment

For the below example, the trained loras are saved in ./loras

# Options
# --path_image : reference image path(s)
# --pfix       : name of LoRA to be saved 
# --prompt     : uninque identifier prompts for each reference image

# For a single style
accelerate launch --num_processes 1 --main_process_port=4401 sag_distribution_alignment.py \
    --path_image assets/references/spatown.png \
    --pfix spatown

# For multiple styles for each identifier prompt
accelerate launch --num_processes 1 --main_process_port=4401 sag_distribution_alignment.py \
    --path_image assets/references/exterior.png assets/references/interior.png \
    --prompt 'exterior' 'interior' \
    --pfix school

Anchor View Generation

For the below example, we provide assets/sampleA as an example. The generated anchor views are saved in ./assets/sampleA/multiviews/spatown_p-e3b0c442_e400_s075157_r12_ip1

# Options
# --pfix            : target lora name
# --epoch           : target lora epoch
# --target          : target input
# --num_replacement : the number of replacements for sparse appearance (warped image)
# --prompt          : additional prompt for style varation
# --offload         : use lower memory

python sag_generate_anchor_view.py --epoch 400 --target assets/sampleA --pfix spatown --num_replacement 12

Geometry-guided Generative Inbetweening (GGI)

Before started, you first download the checkpoint-1350 in here, and move it in ./checkpoints, e.g. , ./checkpoints/checkpoint-1350. The generated video sequence is saved in assets/sampleA/multiviews/spatown_p-e3b0c442_e400_s051106_r12_ip1/d0.5_e1350_n30

# Options
# --target  : target anchor view path
# --offload : use lower memory

python ggi.py --target assets/sampleA/multiviews/spatown_p-e3b0c442_e400_s051106_r12_ip1

📄 Citation

@inproceedings{kim2025videofrom3d,
  author       = {Geonung Kim and Janghyeok Han and Sunghyun Cho},
  title        = {VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Models},
  booktitle    = {SIGGRAPH Asia 2025 Conference Papers (SA Conference Papers '25)},
  year         = {2025},
  address      = {Hong Kong, Hong Kong},
  publisher    = {ACM},
  pages        = {1--11},
  doi          = {10.1145/3757377.3763871},
  isbn         = {979-8-4007-2137-3/25/12},
  url          = {https://doi.org/10.1145/3757377.3763871}
}

☕️ Acknowledgment

• We borrowed the readme format from Upscale-A-Video
• We finetune a pretrained video diffusion model, CogVideoX
• We generate anchor views using Flux ControlNet