Skip to content

distilled_sd.md

Latest commit

 

History

History
118 lines (75 loc) · 5.08 KB

distilled_sd.md

File metadata and controls

118 lines (75 loc) · 5.08 KB

Running distilled models: SSD1B, Vega and SDx.x with tiny U-Nets

Preface

These models feature a reduced U-Net architecture. Unlike standard SDXL models, the SSD-1B and Vega U-Net contains only one middle block and fewer attention layers in its up- and down-blocks, resulting in significantly smaller file sizes. Using these models can reduce inference time by more than 33%. For more details, refer to Segmind's paper: https://arxiv.org/abs/2401.02677v1. Similarly, SD1.x- and SD2.x-style models with a tiny U-Net consist of only 6 U-Net blocks, leading to very small files and time savings of up to 50%. For more information, see the paper: https://arxiv.org/pdf/2305.15798.pdf.

SSD1B

Note that not all of these models follow the standard parameter naming conventions. However, several useful SSD-1B models are available online, such as:

Useful LoRAs are also available:

Vega

Segmind's Vega model is available online here:

VegaRT is an example for an LCM-LoRA:

Both files can be used out-of-the-box, unlike the models described in next sections.

SD1.x, SD2.x with tiny U-Nets

These models require conversion before use. You will need a Python script provided by the diffusers team, available on GitHub:

SD2.x

NotaAI provides the following model online:

Creating a .safetensors file involves two steps. First, run this short Python script to download the model from Hugging Face:

from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained("nota-ai/bk-sdm-v2-tiny",cache_dir="./")

Second, create the .safetensors file by running:

python convert_diffusers_to_original_stable_diffusion.py \
      --model_path  models--nota-ai--bk-sdm-v2-tiny/snapshots/68277af553777858cd47e133f92e4db47321bc74 \
      --checkpoint_path bk-sdm-v2-tiny.safetensors --half --use_safetensors

This will generate the file bk-sdm-v2-tiny.safetensors, which is now ready for use with sd.cpp.

SD1.x

Several Tiny SD 1.x models are available online, such as:

These models also require conversion, partly because some tensors are stored in a non-contiguous manner. To create a usable checkpoint file, follow these simple steps: Download and prepare the model using Python:

Download the model using Python on your computer, for example this way:
import torch
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained("segmind/tiny-sd")
unet=pipe.unet
for param in unet.parameters():
    param.data = param.data.contiguous()     # <- important here
pipe.save_pretrained("segmindtiny-sd", safe_serialization=True)
Run the conversion script:
python convert_diffusers_to_original_stable_diffusion.py \
      --model_path  ./segmindtiny-sd \
      --checkpoint_path ./segmind_tiny-sd.safetensors  --half --use_safetensors

The file segmind_tiny-sd.safetensors will be generated and is now ready for use with sd.cpp. You can follow a similar process for the other models mentioned above.

SDXS-512-DreamShaper

Another very tiny and incredibly fast model is SDXS by IDKiro et al. The authors refer to it as "Real-Time One-Step Latent Diffusion Models with Image Conditions". For details read the paper: https://arxiv.org/pdf/2403.16627 . Once again the authors removed some more blocks of U-Net part and unlike other SD1 models they use an adjusted AutoEncoderTiny instead of default AutoEncoderKL for the VAE part.

Some ready-to-run SDXS-512 model files are available online, such as:
Run the model as follows:
~/stable-diffusion.cpp/build/bin/sd-cli -m sdxs.safetensors -p "portrait of a lovely cat" \
  --cfg-scale 1 --steps 1

Both options: --cfg-scale 1 and --steps 1 are mandatory here.

SDXS-512-0.9

Even though the name "SDXS-512-0.9" is similar to "SDXS-512-DreamShaper", it is completely different but also incredibly fast. Sometimes it is preferred, so try it yourself.

Download a ready-to-run file from here:

For the use of this model, both options --cfg-scale 1 and --steps 1 are again absolutely necessary.