[D] Why is stable diffusion much smaller than predecessors?

Submitted by dahdarknite t3_10r5gku in MachineLearning

Stable diffusion seems to be a departure from the trend of building larger and larger models.

It has 10x less parameters than other image generation models like DALLE-2.

“Incredibly, compared with DALL-E 2 and Imagen, the Stable Diffusion model is a lot smaller. While DALL-E 2 has around 3.5 Billion parameters, and Imagen has 4.6 Billion, the first Stable Diffusion model has just 890 million parameters, which means it uses a lot less VRAM and can actually be run on consumer-grade graphics cards.”

What allows stable diffusion to work so well with a lot less parameters? Are there any drawbacks to this, like requiring stable diffusion to be fine tuned more than DALLE-2 for example?

25

Comments

You must log in or register to comment.

LetterRip t1_j6v57y5 wrote

Mostly the language model - Imagen is using T5-XXL (the 4.6 billion parameters), Dall-E 2 uses GPT-3 (presumably 2.7B not the much larger variants used for ChatGPT). SD is just using CLIP without anything else. The more sophisticated the language model, the better the image generation can understand what you want. CLIP is close to using bag of words.

18

londons_explorer t1_j6wa910 wrote

It's a much smaller model, but IMO, the results are much lower quality too.

However the fact you can run it on your PC means you can tweak all the settings and have many goes at getting better results, partially offsetting that.

2

uhules t1_j6wrx63 wrote

Replying to Mefaso (#1,700,771)

Except DALL-E 2 also applies diffusion in latent space and Imagen performs diffusion in low-res pixel space. My initial hunch was the upscaling diffusion models, but they account for a relatively small portion of the total number of parameters and are more relevant speed-wise. The lackluster explanation is simply "SD does latent better", since you'd need to do an extensive ablation study to compare rather different architectures.

4

Mefaso t1_j6z6zgt wrote

Replying to uhules (#1,702,889)

>DALL-E 2 also applies diffusion in latent space

Not really in the important part. Dalle2 uses diffusion in clip-"latent"-space and then conditions the pixel-diffusion model on the result.

However they still do a full diffusion pass in pixel-space, which is more complex than doing it in latent space, as LDMs do.

1