A Large-Scale Multimodal Dataset and Benchmarks for Human Activity Scene Understanding and Reasoning

_{One recording, seven time-aligned streams: RGB · IR · Thermal · Depth · mmWave Radar · Skeleton · IMU}

📥 Get the data  ·  🛰️ Explore it in your browser  ·  🏆 CUHK-X Challenge @ UbiComp/ISWC 2026 · Grand Finals Oct 12

CUHK-X is a comprehensive multimodal dataset containing 64,267 samples across seven modalities designed for human activity recognition, understanding, and reasoning. It addresses critical gaps in existing HAR datasets by providing synchronized multimodal sensor data with detailed annotations for complex reasoning tasks.

🧭 What's in this repo

Dataset files are distributed separately under a Data Use Agreement. Request access on the dataset portal.

🎉 News

• [Sep 2026] 📦 v1.0.0 is out — first tagged release of the baseline code (SM + LM), project website and Observatory. GitHub now shows a "Cite this repository" button via CITATION.cff.
• [Sep 2026] 🏆 The CUHK-X Challenge is an official competition of UbiComp/ISWC 2026. The Kaggle phase closed on Sep 15; verification and private-data evaluation were scheduled for Sep 19–30. The Grand Finals are scheduled for Oct 12 in Shanghai. See the challenge website for organizer updates. Details in the challenge section below.
• [Jun 2026] 🏆 The CUHK-X Challenge went live on Kaggle. USD $20K prize pool across two tracks (Small Model · Large Model), built on CUHK-X — finals at UbiComp 2026, Shanghai.
• [Jun 2026] 🛰️ The CUHK-X Observatory is live! Stream real CUHK-S clips in the browser: synchronized depth ⊕ infrared ⊕ thermal playback, a 40-action atlas organized by the paper's seven categories, and a playable next-action reasoning quiz with real ground truth.
• [Jun 2026] The project website got a full redesign — now organized into Home, Dataset, Benchmarks & Results and Publications pages.
• [Feb 2026] CUHK-X is accepted by MobiSys 2026!
• [Nov 2025] 🏆 CUHK-X wins the Best Presentation Award at ANAI Workshop @ MobiCom 2025!

🏆 CUHK-X Challenge @ UbiComp/ISWC 2026

The CUHK-X Multimodal Human Activity Challenge is an official competition of UbiComp/ISWC 2026 and the first RGB-free international HAR competition. It runs on Kaggle in two parallel tracks built on CUHK-X, each with an independent USD $10K prize pool (USD $20K total). The baselines in this repository are the reference implementations for both tracks.

Both tracks train on users 1–9 and 16–24 and are scored cross-subject on Kaggle test users 10–11 and 25–26; finalists are additionally evaluated on organizer-held private data. RGB is excluded from the challenge.

Timeline

Grand Finals. Yangtze River Hall, 5F, Shanghai International Convention Center. Small Model Track 09:45–12:30 and Large Model Track 14:30–17:20 (Shanghai time); each team gives a 5-minute talk followed by 5 minutes of Q&A, with remote participation via Zoom. Finalist scores combine the Kaggle private leaderboard (20%), organizer private-data evaluation (30%), reproducibility (10%), technical report (20%), presentation (10%) and model efficiency (10%). Finalist solutions are open-sourced under Apache 2.0 within 30 days of the finals.

Challenge data (as published on the challenge site; RGB excluded, test labels withheld). Small Model Track: Hugging Face · Google Drive · Baidu Netdisk. Large Model Track: Hugging Face · Google Drive · Baidu Netdisk.

Full rules, prizes and certificate tiers: UbiComp/ISWC 2026 competition page · challenge website · questions to cuhkx.competition@gmail.com.

🎯 Key Contributions

• First Multimodal HAU Dataset: CUHK-X is the first dataset to integrate understanding and reasoning across multiple modalities for human action analysis
• Large-Scale & Diverse: 64,267 samples from 30 participants across diverse environments with 7 synchronized modalities
• Novel Evaluation Framework: Three comprehensive benchmarks (HAR, HAU, HARn) with 6 distinct tasks
• LLM-Empowered Annotation: Innovative prompt-based scene creation framework for logical and spatio-temporal representation

📊 Dataset Overview

Modalities (7 Total)

• RGB Video: Standard color video recordings
• Infrared (IR): Thermal imaging for robustness to lighting conditions
• Depth: 3D spatial information from depth cameras
• Thermal: Heat signature analysis
• IMU: Inertial Measurement Unit sensor data
• mmWave Radar: Privacy-preserving motion detection
• Skeleton: 3D pose estimation data

Statistics

• Total Samples: 64,267 annotated action samples
• Participants: 30 diverse subjects
• Environments: 2 indoor environments with varying conditions
• Actions: 40+ different action categories
• Data Types: Both singular actions and sequential activity sequences

🏗️ Dataset Structure

The dataset is organized into two main components:

Small Model Data

• Focus: Singular, well-defined actions (similar to traditional datasets)
• Actions: 40+ different action categories
• Samples: 30,000+ individual action instances (a subset of the 64,267 total samples; the remainder belongs to the sequential Large Model Data)
• Purpose: Traditional HAR evaluation and baseline comparison

Large Model Data

• Focus: Sequential actions performed consecutively
• Purpose: Temporal and emotional analysis, complex reasoning tasks
• Features: Multi-step activity sequences with logical flow
• Applications: Human Action Understanding (HAU) and Next Action Reasoning (HARn)

🎯 Benchmarks & Tasks

1. Human Action Recognition (HAR)

Objective: Traditional action classification across modalities

• Cross-trial evaluation split data with 80% training 20% testing
• Cross-subject evaluation with Leave-One-Subject-Out (LOSO) protocol
• Cross-domain performance analysis different environment data distribution and training results
• Long-tail distribution handling
• Multimodal fusion strategies

2. Human Action Understanding (HAU)

Objective: Comprehend actions through perceptual and contextual integration

Sub-tasks:

1. Action Captioning: Generate natural language descriptions
2. Emotion Analysis: Identify emotional states during activities
3. Sequential Action Reordering: Organize actions chronologically
4. Action Selection: Choose relevant actions from candidates

3. Human Action Reasoning (HARn)

Objective: Infer intentions and causal relationships in action sequences

• Next Action Prediction: Predict likely subsequent actions
• Temporal Reasoning: Understand action progression logic
• Contextual Inference: Consider environmental and situational factors

🔬 Technical Highlights

Novel Framework

• LLM-Generated Scenarios: Consistent and logical activity descriptions
• Human-in-the-Loop Validation: Quality assurance for generated content
• Synchronized Collection: All modalities captured simultaneously
• Environmental Diversity: Multiple settings and conditions

Hardware Setup

• Vzense NYX 650: RGB-D camera for color and depth
• Texas Instruments Radar: mmWave sensing for privacy-preserving detection
• IMU Sensors: Motion and orientation tracking
• Thermal Cameras: Heat signature analysis
• Synchronized Recording: Temporal alignment across all modalities

📈 Key Findings

Model Performance Insights

• Larger models (7B parameters) consistently outperform smaller ones across tasks
• QwenVL-7B and VLLaVA-7B demonstrate superior performance in most benchmarks
• Depth and IR modalities often provide richer information than RGB for reasoning tasks
• Cross-subject performance drops significantly (56.56% vs higher in-domain accuracy)

Challenging Aspects

• Domain Shift: Cross-domain evaluation reveals substantial performance gaps
• Long-tail Distribution: Realistic but challenging class imbalance
• Sequential Reasoning: Complex temporal understanding requires advanced models
• Multimodal Fusion: Optimal combination strategies vary by task

📋 Benchmark Results

HAR Overall Cross-trial Performance

HAU Performance (Selected Tasks)

🎯 Applications

Healthcare & Monitoring

• Cognitive Decline Detection: Identify forgetfulness or repetitive behaviors
• Daily Activity Assessment: Monitor activities of daily living (ADL)
• Rehabilitation Progress: Track recovery through activity analysis

Smart Environments

• Home Automation: Context-aware system responses
• Security & Safety: Anomaly detection in activity patterns
• Human-Computer Interaction: Natural interface design

Research & Education

• Multimodal Learning: Sensor fusion algorithm development
• Temporal Reasoning: Sequential action understanding
• Privacy-Preserving AI: Non-visual sensing research

🏆 Broader Impact

CUHK-X aims to advance research in:

• Conventional HAR: Multimodal algorithms and cross-domain methods
• LLM Evaluation: Benchmark for action understanding capabilities
• Educational Resource: Standard dataset for teaching sensor fusion and multimodal reasoning
• Real-world Deployment: Bridge the gap between lab and practical applications

📝 Citation

If you use CUHK-X in your research, please cite our paper:

@inproceedings{10.1145/3745756.3809209,
  author    = {Jiang, Siyang and Yuan, Mu and Ji, Xiang and Yang, Bufang and Liu, Zeyu and Xu, Lilin and Li, Yang and He, Yuting and Dong, Liran and Lu, Wenrui and Yan, Zhenyu and Jiang, Xiaofan and Gao, Wei and Chen, Hongkai and Xing, Guoliang},
  title     = {A Large-Scale Multimodal Dataset and Benchmarks for Human Activity Scene Understanding and Reasoning},
  booktitle = {Proceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services},
  series    = {MobiSys '26},
  year      = {2026},
  pages     = {352--370},
  numpages  = {19},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  location  = {University of Cambridge, Cambridge, United Kingdom},
  isbn      = {9798400720277},
  doi       = {10.1145/3745756.3809209},
  url       = {https://dl.acm.org/doi/epdf/10.1145/3745756.3809209},
  keywords  = {human action understanding, large language models, datasets}
}

A machine-readable version lives in CITATION.cff; GitHub's "Cite this repository" button uses it.

👥 Contact

For dataset access, questions, or collaborations:

• Email: syjiang [AT] ie.cuhk.edu.hk
• Challenge: cuhkx.competition [AT] gmail.com · UbiComp/ISWC 2026 competition page
• Project Page: CUHK-X

📄 License

Code is released under the MIT License. The dataset is available for non-commercial research under a Data Use Agreement (DUA) and is not redistributable. Our derived annotations/splits are released under CC BY 4.0.

---

Note: This dataset is designed for research and educational purposes. Please ensure compliance with your institution's ethics guidelines when using human activity data.

We obtained approval from an Institutional Review Board (IRB) to conduct this study and collect data from human subjects.