EmoTalk_v0002: Enhanced Speech-Driven Emotional 3D Face Animation

This project is based on the original EmoTalk (ICCV 2023) by Ziqiao Peng et al. Please refer to the original repository for baseline implementation and citation. This fork introduces architectural enhancements and fine-tuning pipelines to improve emotional expressiveness, motion precision, and controllability.

What's New in EmoTalk_v0002?

EmoTalk_v0002 builds upon the original EmoTalk by introducing several key upgrades:

• FiLM-based emotion modulation: enables learnable fusion between emotional and phonetic features.
• GRU-based temporal emotion encoding: captures dynamic emotional variations across time.
• Region-weighted blendshape loss: enhances facial articulation accuracy, especially in lips and brows.
• Level and identity control embeddings: supports controllable emotional intensity and speaker style.
• Editable blendshape output: outputs 52D FLAME-compatible blendshape sequences for real-time animation in Blender.

Research Context

This project was conducted as an academic extension to explore:

• Emotional controllability in speech-driven animation;
• Editable facial animation pipelines compatible with Blender and FLAME;
• Trade-offs in fine-tuning with emotion-less datasets (e.g., HDTF).

Installation

Environment

conda create -n emotalk_v0002 python=3.8
conda activate emotalk_v0002
pip install -r requirements.txt

Blender Setup

wget https://ftp.nluug.nl/pub/graphics/blender/release/Blender3.4/blender-3.4.1-linux-x64.tar.xz
tar -xf blender-3.4.1-linux-x64.tar.xz
mv blender-3.4.1-linux-x64 blender

Demo

demo video on YouTube:

Download the pretrained checkpoint (trained on RAVDESS):

emotalk_finetuned.pth Download link

Place the file under the project root directory.

Then run:

python demo_finetuned.py \
  --wav_path "./audio/online_audio/alive.wav" \
  --result_path "./result/exp_result/7" \
  --model_path "./emotalk_finetuned.pth" \
  --level 7 \
  --person 1

Datasets

This project uses the same datasets as the original EmoTalk work (from the 3DETF dataset):

• RAVDESS — Emotion-annotated speech audio.
• HDTF — High-resolution talking face videos without emotion labels.

The dataset preprocessing scripts are:

• prepare_ravdess.py
• prepare_hdtf.py

Acknowledgements

This project builds upon the excellent work of the original EmoTalk authors. We sincerely thank them for releasing their code and dataset, which made this extension possible.

Credits and dependencies include:

• EmoTalk (ICCV 2023) – Base model architecture and training pipeline.
• 3DETF Dataset – Includes RAVDESS and HDTF subsets for audiovisual facial animation.
• Wav2Vec2 – Pretrained models for speech representation extraction.
• FLAME Model & Blender Integration – Used for 3D mesh reconstruction and animation visualization.