📰 News

• 📄 [2026-08-24] Technical Report — TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming is now available on arXiv!
• 🔥 [2026-08-20] Model Release — TLive-Omni is now available on Hugging Face! With TLive-Omni-4B and TLive-Omni-9B.

📋 Overview

TLive-Omni is an omni-modal understanding model for e-commerce live-stream, mapping image, video, audio, and text into a unified text-output interface. Built on a Qwen3.5 backbone with a grafted AuT audio encoder, it supports 256K tokens of context, trained via a three-stage SFT recipe followed by Faithful-RFT reinforcement fine-tuning.

✨ Highlights

• Timestamped Per-vGrid layout — Audio and video tokens are organized into timestamped grid with explicit boundaries, keeping audio segments adjacent to their corresponding visual content for fine-grained temporal alignment over long streams.
• Three-stage SFT recipe — Progressive training from audio-language alignment to full multimodal SFT, developing live-commerce understanding from omni-modal perception to instruction-following responses.
• Faithful-RFT — A reinforcement fine-tuning stage for faithful and real-time live-stream demands, suppressing explicit reasoning traces and directly optimizing answer quality for live-commerce tasks.
• Rich atomic capabilities — A scenario-oriented taxonomy covering speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, and omni-modal QA, etc, supported by a compact data production engine.
• Strong live-commerce performance with competitive generalization — 4B and 9B variants demonstrate strong results across live-commerce audio, image, and video tasks, together with excellent generalization on general benchmarks.

🏗️ Architecture

TLive-Omni is built on a Qwen3.5 backbone and extends it with a audio encoder through a lightweight MLP aligner, forming a unified text-output omni-modal understanding model. For video inputs with audio, each temporal grid is organized into a timestamped grid that interleaves video and audio token blocks, keeping audio segments adjacent to their corresponding visual content. The model supports up to 256K tokens of context at inference.

📊 Benchmark Results

We evaluate TLive-Omni-4B and TLive-Omni-9B on both live-commerce tasks and general benchmarks. Dash (-) denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold, while the second-best results are in underlined.

Live-Commerce Audio Evaluation

Notes: Live ASR and Spk. ASR denote live-commerce ASR and speaker-attributed ASR.

Live-Commerce Image Evaluation

Live-Commerce Video Evaluation

General Benchmark: Image Reasoning & QA

Notes: MMBench results are reported on the EN-DEV-v1.1 split. VAB and RWQA denote VLMsAreBlind and RealWorldQA.

General Benchmark: Hallucination, OCR, Grounding & Spatial Reasoning

Notes: CharXiv results are on the RQ split.

General Benchmark: Video Understanding

Notes: LongVB denotes LongVideoBench. V-MME and V-MMMU denote Video-MME and VideoMMMU.

General Benchmark: Video Temporal Grounding (TimeLens-Bench)

General Benchmark: Omni-Modal Perception & Reasoning

Notes: OmniVB denotes OmniVideoBench. V-Holmes denotes VideoHolmes.

🎯 Qualitative Examples

Live-Commerce Capabilities

Six representative live-commerce cases:

1. Live-Commerce Video QA — Links audio cues about a hidden 4-cm height boost with visual evidence.
2. Temporal Grounding — Localizes repeated appearances of a queried product badge.
3. Product Visual Grounding — Predicted vs. ground-truth bounding boxes for a queried product.
4. OCR — Text extraction with commerce-oriented semantic labels.
5. Dense Video Captioning — Temporally segmented descriptions of a clothing demonstration.
6. Multi-Dimensional Shot Tagging — Structured labels for shot size, camera, layout and content category.

General-Capability Examples

Six general-capability cases:

1. Dense Video Captioning — Time-aware segment descriptions with visual frames.
2. Temporal Grounding — Localizes a specific action moment with matched intervals.
3. Visual Grounding — Bounding box prediction for queried regions.
4. OCR — Text extraction from nutrition labels at line-level precision.
5. Omni-Modal QA — Cross-modal reasoning combining audio cues with visual evidence.
6. Multi-Dimensional Shot Tagging — Structured labels for shot size, camera, layout and content category.

📦 Model Zoo

⚙️ Installation

This release targets Python 3.10 on Linux x86_64 with CUDA 12.8 and PyTorch 2.10.0.

conda create -n tlive python=3.10 -y
conda activate tlive
pip install -r environments/requirements.txt

environments/requirements.txt includes custom wheels for the supported environment and model. If any wheel does not match your hardware, CUDA version, or Python version, replace it with a compatible build for your setup.

🚀 Quick Start

# Text
python examples/inference_five_modes.py \
  --model /path/to/model --mode text

# Image
python examples/inference_five_modes.py \
  --model /path/to/model --mode image --image data/image.jpg

# Standalone audio
python examples/inference_five_modes.py \
  --model /path/to/model --mode audio --audio data/audio.mp3

# Video frames + audio track
python examples/inference_five_modes.py \
  --model /path/to/model --mode vocal-video --video data/vocal_video.mp4

# Video frames only
python examples/inference_five_modes.py \
  --model /path/to/model --mode silence-video --video data/silence_video.mp4

For temporal localization outputs, we recommend the MM:SS - MM:SS interval format, for example 01:23 - 01:35.

For direct processor calls, set use_audio_in_video=True to use the video's audio track, or False for visual-only video:

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    use_audio_in_video=True,
)

⚡ vLLM

Installation

First install the pre-built wheel (Python 3.10 + CUDA 12.8 + Linux x86_64), built and tested on NVIDIA H20 GPUs (Hopper, sm_90):

pip install https://github.com/TaoLiveAIGC/TLive-Omni/releases/download/v1.0.0-rc1/vllm-0.19.0+cu128-cp310-cp310-linux_x86_64.whl

If your GPU, driver, or CUDA setup is not compatible with this wheel, build vLLM from source using the customized code in vllm/.

Inference

# Text
python examples/inference_vllm_five_modes.py \
  --model /path/to/model --mode text

# Image
python examples/inference_vllm_five_modes.py \
  --model /path/to/model --mode image --image data/image.jpg

# Standalone audio
python examples/inference_vllm_five_modes.py \
  --model /path/to/model --mode audio --audio data/audio.mp3

# Video frames + audio track
python examples/inference_vllm_five_modes.py \
  --model /path/to/model --mode vocal-video --video data/vocal_video.mp4

# Video frames only
python examples/inference_vllm_five_modes.py \
  --model /path/to/model --mode silence-video --video data/silence_video.mp4

🙏 Acknowledgement

TLive-Omni is built with reference to the following open-source projects: Qwen3.5, Qwen3-Omni, Transformers, ms-swift, DeepSpeed, and vLLM. We sincerely thank these projects and the Qwen team for their outstanding open-source models.

📖 Citation

If you find our work helpful, please consider citing our paper:

@article{hu2026tliveomni,
  title={TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming},
  author={Hu, Yibo and Qian, Yu and Gu, Mao and Tao, Yingfan and Chen, Yuhao and Luo, Yongdong and Liu, Zhuoqun and Jin, Meiguang and Ma, Junfeng},
  journal={arXiv preprint arXiv:2608.20958},
  year={2026}
}

📄 License

This project is released under the Apache License 2.0.