OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation

🌍 Useful Links

• Project Page: http://correr-zhou.github.io/OmniShow
• Paper: https://arxiv.org/abs/2604.11804
• Code: https://github.com/Correr-Zhou/OmniShow

📌 Benchmark Summary

HOIVG-Bench is a multimodal benchmark for Human-Object Interaction Video Generation (HOIVG). It is designed to evaluate how well a model can jointly align text prompts, human reference images, object reference images, audio signals, and pose conditions when synthesizing human-object interaction videos.

Unlike existing benchmarks that mainly focus on limited-modality control, HOIVG-Bench emphasizes the synergy among multiple conditioning signals required by practical HOIVG scenarios. Each sample is organized as paired benchmark data with a single CSV entry file and relative paths to all associated modalities.

🗂️ File Structure

The released benchmark is organized as follows:

HOIVG-Bench/
├── meta_data.csv
├── ref_image_human/
├── ref_image_object/
├── audio/
├── pose_video/
└── pose_data/

• meta_data.csv: entry file that indexes all benchmark samples.
• ref_image_human/: human reference images.
• ref_image_object/: object reference images.
• audio/: speech audio files aligned with the target interaction.
• pose_video/: pose videos used as motion guidance.
• pose_data/: processed pose annotations stored as PyTorch .pt files.

🧾 Data Fields

Each row in meta_data.csv contains the following fields:

🔍 Example Sample

An example entry from meta_data.csv is shown below:

{
  "text_prompt": "The young woman with light brown hair pulled back stands in a brightly lit room with white walls, holding a small, light green dropper bottle labeled HOIVG-Bench. The product is a small-size bottle, compact enough to be comfortably held in one hand. She gestures expressively with her hands while demonstrating and speaking directly to the camera, showing how to use the bottle and engaging the viewer with her presentation.",
  "ref_image_human": "ref_image_human/00000_video_0_human.png",
  "ref_image_object": "ref_image_object/00000_video_0_object.png",
  "object_label": "bottle",
  "audio": "audio/00000_video_0_audio.wav",
  "audio_caption": "Discover the power of this gentle serum, designed to nourish and refresh your skin. Its smooth texture absorbs quickly, leaving a natural, healthy glow. Perfect for daily use, this lightweight formula fits seamlessly into your skincare routine.",
  "pose_video": "pose_video/00000_video_0_pose.mp4",
  "pose_data": "pose_data/00000_video_0_pose_data.pt"
}

🎯 Intended Uses

HOIVG-Bench is intended for benchmarking and analysis of multimodal human-object interaction video generation systems. Typical use cases include:

• Evaluating how well a model preserves human and object reference consistency.
• Measuring condition alignment under text, audio, and pose guidance.
• Comparing models across multiple HOIVG task settings.
• Studying the interaction between visual, motion, and speech conditions in generated videos.

This release is primarily intended for research and benchmarking, rather than unrestricted training use.

🧩 Benchmark Tasks

HOIVG-Bench supports evaluation under the following multimodal settings:

• Reference-to-Video (R2V): text + reference images -> video
• Reference+Audio-to-Video (RA2V): text + reference images + audio -> video
• Reference+Pose-to-Video (RP2V): text + reference images + pose -> video
• Reference+Audio+Pose-to-Video (RAP2V): text + reference images + audio + pose -> video

These settings are designed to cover both standard and challenging multimodal generation scenarios. In particular, the full RAP2V setting evaluates a model's ability to jointly orchestrate all available modalities in a unified generation pipeline.

📏 Evaluation Protocol

HOIVG-Bench follows the evaluation setting described in the OmniShow paper.

• For fair comparison, all quantitative metrics and qualitative analyses are standardized to 5-second video clips at 720p resolution in portrait mode.
• Evaluation covers five key dimensions:
  • Text Alignment: TA
  • Reference Consistency: FaceSim, NexusScore
  • Audio-Visual Synchronization: Sync-C, Sync-D
  • Pose Accuracy: AKD, PCK
  • Video Quality: AES, IQA, VQ, MQ
• Different task settings use the metrics applicable to their available modalities.

The official evaluation code is currently under internal review and will be released later.

💻 How to Load

You can load the benchmark metadata with standard Python tools such as pandas.

from pathlib import Path
import pandas as pd

root = Path("HOIVG-Bench")
meta = pd.read_csv(root / "meta_data.csv")

print(meta.columns.tolist())
sample = meta.iloc[0].to_dict()
print(sample)

human_image = root / sample["ref_image_human"]
object_image = root / sample["ref_image_object"]
audio_path = root / sample["audio"]
pose_video_path = root / sample["pose_video"]
pose_data_path = root / sample["pose_data"]

If you want to inspect the pose tensor:

import torch

pose_data = torch.load(pose_data_path, map_location="cpu")
print(type(pose_data))

🏗️ Data Construction

HOIVG-Bench is constructed through a rigorous multimodal curation pipeline:

1. Video Curation: Raw videos are selected from an internal video library. Candidate videos must contain clear human-object interactions, exceed 4 seconds in duration, and cover diverse human attributes and object categories.
2. Object Image Acquisition: Instead of directly cropping objects from source videos, we create high-quality reference object images by modifying textures and colors while preserving fine-grained details.
3. Human Image Acquisition: For privacy protection and identity de-identification, human reference images are generated from video screenshots while maintaining stylistic similarity to the original subjects.
4. Pose Extraction: Per-frame human pose skeletons are extracted from the original videos and provided as motion guidance signals.
5. Audio Synthesis: Semantically aligned speech audio is constructed through a two-stage process, where scripts are first generated for the target object and then synthesized with matched voice attributes.

To support public release, the use of generated human and object reference images is a deliberate design choice motivated by privacy, ethical, and legal considerations. Additional manual checks are performed to reduce noticeable synthetic artifacts and mitigate domain bias.

⚖️ Ethics

This dataset is released for research purposes only.

• Human reference images are generated with identity de-identification in mind.
• The release is designed to reduce privacy, ethical, and legal risks associated with public benchmark distribution.
• The dataset should not be used for identity misuse, deception, harassment, or harmful content generation.

🔗 Citation

If you find this benchmark useful in your research, please cite:

@article{zhou2026omnishow,
  title={OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation},
  author={Zhou, Donghao and Liu, Guisheng and Yang, Hao and Li, Jiatong and Lin, Jingyu and Huang, Xiaohu and Liu, Yichen and Gao, Xin and Chen, Cunjian and Wen, Shilei and Fu, Chi-Wing and Heng, Pheng-Ann},
  journal={arXiv preprint arXiv:2604.11804},
  year={2026}
}

📬 Contact

For questions about the benchmark, please feel free to email Donghao Zhou: dhzhou@link.cuhk.eduk.hk.