Video Diffusion: Stopping the Flicker

Dec 30, 2025 • 20 min read

The biggest challenge in AI video generation isn't resolution — it's temporal consistency. If Frame 1 shows a red shirt and Frame 2 shows a blue shirt, the video flickers. If the background subtly changes between frames, it strobes. Your brain immediately recognizes this as artificial. Solving temporal consistency is what separates toy demos from production-quality video content. This guide covers the two dominant open-source approaches — Stable Video Diffusion for image-to-video and AnimateDiff for text-to-video — along with post-processing techniques to smooth the output.

1. Stable Video Diffusion (SVD): Image to Motion

SVD takes a single image and hallucinates 2-4 seconds of realistic motion. It was trained on millions of video clips to understand how real-world objects move:

pip install diffusers transformers accelerate

from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import load_image, export_to_video
import torch

# Load SVD-XT (21 frames, more motion) or SVD (25 frames, smoother)
pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt",
    torch_dtype=torch.float16,
    variant="fp16",
)
pipe.enable_model_cpu_offload()   # Reduce VRAM usage (8GB+ GPU needed)
pipe.unet.enable_forward_chunking()  # Process frames in chunks for low VRAM

# Load starting frame (resize to 1024x576 for optimal results)
from PIL import Image
image = load_image("./product_photo.jpg").resize((1024, 576))

# Generate video
frames = pipe(
    image,
    num_frames=21,             # 21 frames = ~2.3s at 9fps
    decode_chunk_size=8,       # Decode 8 frames at a time (VRAM tradeoff)
    generator=torch.manual_seed(42),
    
    # Key parameters for controlling motion:
    motion_bucket_id=127,      # 1-255: higher = more motion (default 127)
    fps=9,                     # Target FPS during generation
    noise_aug_strength=0.02,   # Add slight noise for natural look (0.0-0.1)
).frames[0]

export_to_video(frames, "output.mp4", fps=9)

# Limitations of SVD:
# - Low control: you can't specify "move left" or "character walks"
# - Short: max 4 seconds per generation (chain multiple for longer)
# - No text conditioning: image-only input
# 
# SVD excels at: product/food photography animation, natural phenomena
# (water, fire, clouds, plants blowing in wind)

2. AnimateDiff: Text-to-Video with Motion Modules

from diffusers import AnimateDiffPipeline, LCMScheduler, MotionAdapter
from diffusers.utils import export_to_gif, export_to_video
import torch

# AnimateDiff: injects motion awareness into ANY SD 1.5 checkpoint
# The "Motion Module" is a separate adapter that was trained on video data

# Load motion adapter (handles temporal consistency between frames)
adapter = MotionAdapter.from_pretrained(
    "guoyww/animatediff-motion-adapter-v1-5-3",
    torch_dtype=torch.float16,
)

# Use your favorite SD 1.5 base model (all LoRAs still work!)
pipe = AnimateDiffPipeline.from_pretrained(
    "emilianJR/epiCRealism",  # Any SD1.5 checkpoint works
    motion_adapter=adapter,
    torch_dtype=torch.float16,
)

# LCM for fast sampling (4x faster than standard DDIM)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config, beta_schedule="linear")

pipe.load_lora_weights("guoyww/animatediff-motion-lora-zoom-in", adapter_name="zoom-in")
pipe.set_adapters(["zoom-in"], [0.8])  # Apply zoom-in motion LoRA at 80% strength

pipe.enable_vae_slicing()
pipe.enable_model_cpu_offload()

output = pipe(
    prompt="professional product photography of a luxury watch, dramatic lighting, slow rotation, cinematic",
    negative_prompt="blurry, distorted, flickering, artifacts",
    num_frames=16,           # 16 frames = ~2s at 8fps
    guidance_scale=7.5,
    num_inference_steps=20,  # With LCM scheduler, 4-8 steps sufficient
    generator=torch.manual_seed(42),
    width=512, height=512,   # SD1.5 native resolution
)

export_to_video(output.frames[0], "watch_animation.mp4", fps=8)

# AnimateDiff strengths vs SVD:
# ✓ Full text control over content and style
# ✓ Works with ALL SD 1.5 checkpoints and LoRAs (dreamshaper, revAnimated, etc.)
# ✓ Motion LoRAs: zoom-in, zoom-out, pan-left, pan-right, roll, tilt
# ✓ Prompt scheduling: different prompt for different frame ranges
# ✗ Lower quality than SVD for photorealistic subjects
# ✗ Limited to SD1.5 ecosystem (not SDXL)

3. Controlled Video: ControlNet + AnimateDiff

# AnimateDiff + ControlNet: control character pose across all frames
# Use a reference video's pose/depth as control signal

from controlnet_aux import OpenposeDetector
from diffusers import ControlNetModel

# Extract poses from reference video using OpenPose
pose_detector = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")

def extract_video_poses(video_path: str) -> list:
    """Extract OpenPose skeleton from each frame of a reference video."""
    cap = cv2.VideoCapture(video_path)
    poses = []
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        pose_image = pose_detector(Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)))
        poses.append(pose_image)
    cap.release()
    return poses

reference_poses = extract_video_poses("./person_walking.mp4")  # 16 frames

# Load ControlNet for OpenPose with AnimateDiff
controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/control_v11p_sd15_openpose",
    torch_dtype=torch.float16,
)

# Generate video conditioned on poses from reference video
output = pipe_with_controlnet(
    prompt="anime character walking through a futuristic city, neon lights, rain",
    controlnet_conditioning_image=reference_poses,  # Use reference poses
    controlnet_conditioning_scale=0.7,              # How strongly to follow poses
    num_frames=16,
    guidance_scale=7.5,
    num_inference_steps=20,
)
# Result: AI character walks exactly like the reference video
# but with completely different style, environment, and appearance

4. RIFE Frame Interpolation: From 8fps to 60fps

# RIFE (Real-Time Intermediate Flow Estimation) interpolates frames
# Hallucinates smooth motion between generated keyframes
# 8fps → 60fps in seconds

pip install rife-ncnn-vulkan  # Cross-platform, uses Vulkan GPU

# Command-line interpolation:
rife-ncnn-vulkan -i input_8fps.mp4 -o output_60fps.mp4 \
    -m rife-v4.6 \          # Use latest RIFE model
    -n 8                     # 8x interpolation (8fps × 8 = 64fps, output as 60fps)
    -f mp4 \
    -s 1.0                   # Speed factor (1.0 = same speed)

# Python API:
from rife_ncnn_vulkan import Rife

rife = Rife(gpuid=0, model="rife-v4.6")

# Interpolate between two specific frames
frame1 = Image.open("frame_0008.png")
frame2 = Image.open("frame_0009.png")

# Get 7 intermediate frames (effectively 8x interpolation)
intermediate_frames = rife.interpolate(frame1, frame2, n=7)
# output: [frame1, interp1, interp2, ..., interp7, frame2]

# Warning: RIFE can introduce artifacts for:
# - Very fast motion (blur instead of crisp interpolation)
# - Sudden scene cuts
# - Objects that teleport between frames
# → Use AnimateDiff's "sliding window" for proper temporal consistency first

Frequently Asked Questions

When should I use SVD vs AnimateDiff?

Use SVD when you have a photorealistic starting image and want realistic natural motion (water, smoke, wind, gentle character animation). SVD produces higher quality photorealism but gives you little control over motion direction. Use AnimateDiff when you need text control (specifying content, style, and motion type), when you want to leverage the huge SD 1.5 ecosystem of LoRAs and checkpoints, or when you need stylized/anime aesthetics. For production, consider using SVD for close-up product shots and AnimateDiff for narrative/character-driven content.

How do I generate videos longer than 4 seconds?

Chain multiple generations: use the last frame of video chunk N as the first frame of video chunk N+1 for SVD. For AnimateDiff, use the "sliding window" approach where each generation overlaps with the previous by 4-8 frames. The overlap ensures seamless transitions. For 30+ second videos, consider text-to-video platforms like Runway Gen-3 or Kling AI that have been specifically optimized for long-form generation.

Conclusion

The video diffusion ecosystem split into two clear camps: SVD for high-quality image-to-video with photorealistic motion, and AnimateDiff for text-controllable video with full access to the SD 1.5 model ecosystem. Both produce 6-9 FPS raw output that benefits from RIFE interpolation to reach 24-60 FPS for playback. ComfyUI is the optimal interface for building complex video generation workflows, combining these models with ControlNet, IP-Adapter, and post-processing nodes in a visual pipeline.