SDXL-VAE-FP16-Fix

SDXL-VAE-FP16-Fix is the SDXL VAE*, but modified to ru i fp16 precisio without geeratig NaNs.

VAE	Decodig i float32 / bfloat16 precisio	Decodig i float16 precisio
SDXL-VAE	✅	⚠️
SDXL-VAE-FP16-Fix	✅	✅

? Diffusers Usage

Just load this checkpoit via AutoecoderKL:

import torch
from diffusers import DiffusioPipelie, AutoecoderKL
from modelscope import sapshot_dowload

local_vae = sapshot_dowload("AI-ModelScope/sdxl-vae-fp16-fix",revisio='master')
local_sd = sapshot_dowload("AI-ModelScope/stable-diffusio-xl-base-1.0",revisio='master')
local_refier = sapshot_dowload("AI-ModelScope/stable-diffusio-xl-refier-1.0",revisio='master')
vae = AutoecoderKL.from_pretraied(local_vae, torch_dtype=torch.float16)
pipe = DiffusioPipelie.from_pretraied(local_sd, vae=vae, torch_dtype=torch.float16, variat="fp16", use_safetesors=True)
pipe.to("cuda")
refier = DiffusioPipelie.from_pretraied(local_refier, vae=vae, torch_dtype=torch.float16, use_safetesors=True, variat="fp16")
refier.to("cuda")

_steps = 40
high_oise_frac = 0.7

prompt = "A majestic lio jumpig from a big stoe at ight"

image = pipe(prompt=prompt, um_iferece_steps=_steps, deoisig_ed=high_oise_frac, output_type="latet").images
image = refier(prompt=prompt, um_iferece_steps=_steps, deoisig_start=high_oise_frac, image=image).images[0]
image

Details

SDXL-VAE geerates NaNs i fp16 because the iteral activatio values are too big:

SDXL-VAE-FP16-Fix was created by fietuig the SDXL-VAE to:

1. keep the fial output the same, but
2. make the iteral activatio values smaller, by
3. scalig dow weights ad biases withi the etwork

There are slight discrepacies betwee the output of SDXL-VAE-FP16-Fix ad SDXL-VAE, but the decoded images should be close eough for most purposes.

* sdxl-vae-fp16-fix is specifically based o SDXL-VAE (0.9), but it works with SDXL 1.0 too

SDXL-VAE-FP16-Fix SDXL-VAE-FP16-Fix is the SDXL VAE*, but modified to run in fp16 precision without