AVTR-1 is a flow-matching-based autoregressive model for live dialogue. Given a portrait image and dual-stream audio, it renders lip-synced speech and active listening at 25 fps on a single GPU. Built for production deployment: model weights, TensorRT-accelerated inference, and the live-session backend - available as an API or fully self-hosted

📑 What's included

• Model weights
• Inference code
• Interactive streaming demo
• Technical report (Coming soon)
• Production-ready back-end (Coming soon)

Table of Contents

1. Quick Start
2. Performance
3. Troubleshooting

1. 🚀 Quick Start

Prerequisites

• Linux
• NVIDIA GPU (Ampere or later recommended)
• CUDA 12.x + TensorRT 10.x
• pixi — curl -fsSL https://pixi.sh/install.sh | sh

Install

git clone https://github.com/avaturn-live/avtr-1.git
cd avtr-1
pixi install

Set storage path (Optional)

export AVTR1_LOCAL_STORAGE=/path/to/avtr1_storage

All downloaded weights and built engines go here. Defaults to <project_root>/artifacts/ (the repo checkout, not the caller's working directory) when unset.

Download weights

pixi run download

First run will prompt for a HuggingFace login via hf auth login (automatically invoked as a dependency of download).

Build TRT engines

Weights are pulled from two public HF repos by the previous step: AVTR-1 weights from avaturn-live/avtr-1 and LivePortrait weights repackaged as ONNX graphs from digital-avatar/ditto-talkinghead. TRT engines are compute-capability specific and built locally — run the scripts below once per machine; outputs land under $AVTR1_LOCAL_STORAGE.

# Build everything at once
pixi run build-trt-engines

# Or individually
pixi run build-trt-engines-avtr1
pixi run build-trt-engines-renderer
pixi run build-trt-engines-hubert

Run interactive demo

pixi run interactive-demo

Run offline generation

Single speaker. Avatar lip-syncs the given audio track.

pixi run generate_offline --speech example/speaker_1.ogg

# with a custom avatar and background:
pixi run generate_offline --speech example/speaker_1.ogg --avatar maria --bg minimal_office

Two-speaker dialogue. Avatar voices --speech and reacts (active listening) to the peer audio on --listen. Run twice with the tracks swapped to render both sides of the conversation.

# avatar = speaker 1 (elena)
pixi run generate_offline --speech example/speaker_1.ogg --listen example/speaker_2.ogg --avatar elena  --out elena.mp4
# avatar = speaker 2 (marcus)
pixi run generate_offline --speech example/speaker_2.ogg --listen example/speaker_1.ogg --avatar marcus --out marcus.mp4

# stitch both sides into a single side-by-side video:
ffmpeg -i elena.mp4 -i marcus.mp4 -filter_complex \
  "[0:v][1:v]hstack=inputs=2[v];[0:a][1:a]amix=inputs=2[a]" \
  -map "[v]" -map "[a]" dialogue.mp4

Silence / idle motion. No audio — renders idle micro-motion for the given duration.

pixi run generate_offline --duration 10

Available avatars are the filenames (without .png) inside $AVTR1_LOCAL_STORAGE/v1/avatars_artifacts/reference_frames/ after downloading.

2. Performance

Per-chunk latency

AVTR-1 generates motion in 5-frame chunks end-to-end. At 25 fps that's 200 ms of output per chunk, so any GPU under that line runs in real-time.

Real-time factor = 200 ms / latency. ≥ 1.0× means the GPU keeps up with 25 fps.

Troubleshooting

CLOUDFLARE_TURN_KEY_ID="<Turn Key ID>"
CLOUDFLARE_TURN_KEY_TOKEN="<API Token>"

License

This repository contains three separately licensed components:

• scripts/ — build and demo tooling, released under the AVTR-1 Community License (LICENSE-MODEL.md). Permits commercial use by entities under USD 10M annual revenue; entities at or above that threshold need a commercial agreement. The same license governs the AVTR-1 weights distributed at avaturn-live/avtr-1.
• src/avtr1_renderer/ — Avaturn Renderer (inference pipeline), released under the PolyForm Noncommercial License 1.0.0 with a Required Notice (LICENSE-RENDERER.md). Noncommercial use only, regardless of revenue; any commercial use needs a separate Renderer Commercial License.
• src/avaturn_live_streamer/ — Avaturn Streamer (orchestration backend), released under the PolyForm Noncommercial License 1.0.0 with a Required Notice and patent reservation (LICENSE-STREAMER.md, PATENTS.md). Noncommercial use only, regardless of revenue; any commercial use needs a separate Streamer Commercial License.

See LICENSE.md for the full component map and the consequences of the multi-license structure. In any conflict between this summary and the underlying license files, the license files control.

Non-commercial dependency

The pipeline uses InsightFace's pretrained SCRFD detector and 2D106 landmark model, which are licensed for non-commercial research use only. To use AVTR-1 commercially you must either obtain a commercial license from InsightFace (deepinsight@gmail.com) or replace these models with permissively-licensed alternatives (e.g., MediaPipe). See THIRD-PARTY-NOTICES.md for the full picture.

Commercial inquiries: hello@avaturn.me