Motion LoRAs

Motion LoRAs allow adding specific types of motion to your animations.

Currently the following types of motion are available for models using the guoyww/animatediff-motion-adapter-v1-5-2 checkpoint.

• Zoom In/Out
• Pan Left/Right
• Tilt Up/Down
• Rolling Clockwise/Anticlockwise

Please refer to the AnimateDiff documentation for information on how to use these Motion LoRAs.

Install requirements:

pip install transformers peft diffusers -U

import torch
from diffusers import MotionAdapter, AnimateDiffPipeline, DDIMScheduler
from diffusers.utils import export_to_gif
from modelscope import snapshot_download

model_dir = snapshot_download("Shanghai_AI_Laboratory/animatediff-motion-adapter-v1-5-2")
# Load the motion adapter
adapter = MotionAdapter.from_pretrained(model_dir)
# load SD 1.5 based finetuned model
model_id = snapshot_download("wyj123456/Realistic_Vision_V5.1_noVAE")
pipe = AnimateDiffPipeline.from_pretrained(model_id, motion_adapter=adapter)
lora_dir = snapshot_download("Shanghai_AI_Laboratory/animatediff-motion-lora-rolling-clockwise")
pipe.load_lora_weights(lora_dir, adapter_name="rolling-clockwise")
pipe.set_adapters(["rolling-clockwise"], adapter_weights=[1.0])
scheduler = DDIMScheduler.from_pretrained(
    model_id, subfolder="scheduler", clip_sample=False, timestep_spacing="linspace", steps_offset=1
)
pipe.scheduler = scheduler

# enable memory savings
pipe.enable_vae_slicing()
pipe.enable_model_cpu_offload()

output = pipe(
    prompt=(
        "masterpiece, bestquality, highlydetailed, ultradetailed, sunset, "
        "orange sky, warm lighting, fishing boats, ocean waves seagulls, "
        "rippling water, wharf, silhouette, serene atmosphere, dusk, evening glow, "
        "golden hour, coastal landscape, seaside scenery"
    ),
    negative_prompt="bad quality, worse quality",
    num_frames=16,
    guidance_scale=7.5,
    num_inference_steps=25,
    generator=torch.Generator("cpu").manual_seed(42),
)
frames = output.frames[0]
export_to_gif(frames, "animation.gif")