Dippy-WAN: Animated Charades for Language Learning

Generate loopable animation clips where a character acts out sentences. Each sentence produces a forward pass (action) and a reset pass (return to neutral), creating seamless clips that chain into trajectories.

Example

Avatar Image → "He jumped" → forward clip → reset clip → loop
                                                          ↓
             → "She waved" → forward clip → reset clip → chain...

Quick Start

pip install diffusers==0.36.0 transformers==5.1.0 accelerate==1.12.0 \
  huggingface_hub==1.4.1 gradio==6.5.1 \
  spaces ftfy peft imageio-ffmpeg opencv-python safetensors sentencepiece openai

python dippy-app.py

Or open Dippy_WAN.ipynb in Google Colab.

Backends

Select via DIPPY_BACKEND=wan14b env var or the Gradio dropdown.

API

python api.py

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"sentence": "He jumped", "image_path": "avatars/Perplexity.png"}'

Key Files

How It Works

The reset pass uses WAN's last_image parameter to condition on both the action pose (first frame) and the original avatar (last frame), producing a smooth return to neutral without a hard cut.

Clips are cached by (sentence, backend, avatar_hash) for instant replay.

License

Research use. See model licenses: WAN 2.1, CogVideoX.