MIntRec2.0

Features • Download • Dataset Description • Benchmark Framework • Quick start

MIntRec2.0 is a large-scale multimodal multi-party benchmark dataset for intent recognition and out-of-scope detection in conversations. We also provide benchmark framework and evaluation codes for usage.

Example:

Updates 🔥 🔥 🔥

---

Features

MIntRec2.0 has the following features:

• Large in Scale: Compared with our first version of multimodal intent recognition dataset (MIntRec), MIntRec2.0 increase the data-scale from 2.2K to 15K, with 30 intent classes, 9.3K in-scope and 5.7K out-of-scope annotated utterances with text, video, and audio modalities.

• Multi-turn & Multi-party Dialogues: It contains 1,245 dialogues with an average of 12 utterances per dialogue in continuous conversations. Each utterance has an intent label in each dialogue. Each dialogue has at least two different speakers with annotated speaker identities for each utterance.

• Out-of-scope Detection: As real-world dialogues are in the open-world scenarios as suggested in TEXTOIR, we further include an OOS tag for detecting those utterances that do not belong to any of existing intent classes. They can be used for out-of-distribution detection and improve system robustness.

Download

Feature and raw videos

We provide full feature (video and audio), text annotations, and raw videos of in-scope (13G) and out-of-scope data (7.44G), which can be downloaded from Google Drive.

Dataset Description

• Data sources: The raw videos are collected from three TV series: Superstore, The Big Bang Theory, and Friends.
• Dialogue division: We manually divide dialogues based on the scenes and episode.
• Speaker information: We manually annotate 21, 7, 6 main characters in Superstore, The Big Bang Theory, and Friends, respectively.
• Intent classes
  • Express emotions or attitudes (16): doubt, acknowledge, refuse, warn, emphasize, complain, praise, apologize, thank, criticize, care, agree, oppose, taunt, flaunt, joke
  • Acheve goals (14): ask for opinions, confirm, explain, invite, plan, inform, advise, arrange, introduce, comfort, leave, prevent, greet, ask for help

Statistics

Data distribution of in-scope (IS) and out-of-scope (OOS) samples:

Intent distribution:

Benchmark Framework

We present a framework to benchmark multimodal intent understanding and out-of-scope detection in both single-turn and multi-turn conversational scenarios.

The overall framework:

The framework contains 4 main modules:

• Data Organization: Single-turn dialogues use utterance-level samples as inputs. Multi-turn dialogues are arranged chronologically based on the order in which the speakers take their turn.
• Multimodal Feature Extraction: Extracting features from text, video, and audio modalities. For multi-turn dialogues, we concatenate the context information with the current utterance and separate them with a special token.
• Multimodal Fusion: Multimodal fusion methods (e.g., MAG-BERT, MulT) can be used for fusing different modalities.
• Training: In-scope data uses cross-entropy loss. Out-of-scope data uses outlier exposure loss. It may also contain the multimodal fusion loss for capturing cross-modal interactions.
• Inference: Open set recognition method (e.g., DOC) can be used to identify K known classes and detect one out-of-scope class.

Quick start

1. Use anaconda to create Python environment

   conda create --name MIntRec python=3.9
   conda activate MIntRec
   

2. Install PyTorch (Cuda version 11.2)

   conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
   

3. Clone the MIntRec repository.

   git clone git@github.com:thuiar/MIntRec2.0.git
   cd MIntRec
   

4. Install related environmental dependencies

   pip install -r requirements.txt
   

5. Run examples (Take mag-bert as an example, more can be seen here)

   sh examples/run_mag_bert_baselines.sh
   

Citations

If this work is helpful, or you want to use the codes and results in this repo, please cite the following papers:

• MIntRec2.0: A Large-scale Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations
• MIntRec: A New Dataset for Multimodal Intent Recognition

@inproceedings{
  zhang2024mintrec,
  title={{MI}ntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations},
  author={Hanlei Zhang and Xin Wang and Hua Xu and Qianrui Zhou and Kai Gao and Jianhua Su and jinyue Zhao and Wenrui Li and Yanting Chen},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2024},
  url={https://openreview.net/forum?id=nY9nITZQjc}
}

@inproceedings{MIntRec,
   author = {Zhang, Hanlei and Xu, Hua and Wang, Xin and Zhou, Qianrui and Zhao, Shaojie and Teng, Jiayan},
   title = {MIntRec: A New Dataset for Multimodal Intent Recognition},
   year = {2022},
   booktitle = {Proceedings of the 30th ACM International Conference on Multimedia},
   pages = {1688–1697},
}

The dataset and camera ready version of the paper will be updated recently.