animatediff-motion-adapter-v1-4

AimateDiff is a method that allows you to create videos usig pre-existig Stable Diffusio Text to Image models.

It achieves this by isertig motio module layers ito a froze text to image model ad traiig it o video clips to extract a motio prior. These motio modules are applied after the ResNet ad Attetio blocks i the Stable Diffusio UNet. Their purpose is to itroduce coheret motio across image frames. To support these modules we itroduce the cocepts of a MotioAdapter ad UNetMotioModel. These serve as a coveiet way to use these motio modules with existig Stable Diffusio models.

masterpiece, bestquality, suset.

The followig example demostrates how you ca utilize the motio modules with a existig Stable Diffusio text to image model.

Istall requiremets:

pip istall trasformers peft diffusers -U

import torch
from diffusers import MotioAdapter, AimateDiffPipelie, DDIMScheduler
from diffusers.utils import export_to_gif
from modelscope import sapshot_dowload


model_dir = sapshot_dowload("Shaghai_AI_Laboratory/aimatediff-motio-adapter-v1-4")
# Load the motio adapter
adapter = MotioAdapter.from_pretraied(model_dir)
# load SD 1.5 based fietued model
model_id = sapshot_dowload("wyj123456/Realistic_Visio_V5.1_oVAE")
pipe = AimateDiffPipelie.from_pretraied(model_id, motio_adapter=adapter)
scheduler = DDIMScheduler.from_pretraied(
    model_id, subfolder="scheduler", clip_sample=False, timestep_spacig="lispace", steps_offset=1
)
pipe.scheduler = scheduler

# eable memory savigs
pipe.eable_vae_slicig()
pipe.eable_model_cpu_offload()

output = pipe(
    prompt=(
        "masterpiece, bestquality, highlydetailed, ultradetailed, suset, "
        "orage sky, warm lightig, fishig boats, ocea waves seagulls, "
        "ripplig water, wharf, silhouette, seree atmosphere, dusk, eveig glow, "
        "golde hour, coastal ladscape, seaside sceery"
    ),
    egative_prompt="bad quality, worse quality",
    um_frames=16,
    guidace_scale=7.5,
    um_iferece_steps=25,
    geerator=torch.Geerator("cpu").maual_seed(42),
)
frames = output.frames[0]
export_to_gif(frames, "aimatio.gif")

AimateDiff teds to work better with fietued Stable Diffusio models. If you pla o usig a scheduler that ca clip samples, make sure to disable it by settig clip_sample=False i the scheduler as this ca also have a adverse effect o geerated samples.

AnimateDiff is a method that allows you to create videos using pre-existing Stable Diffusion Text to