Dataset Description, Collection, and Source

MCIF (Multimodal Crosslingual Instruction Following) is a multilingual human-annotated benchmark based on scientific talks that is designed to evaluate instruction-following in crosslingual, multimodal settings over both short- and long-form inputs. MCIF spans three core modalities -- speech, vision, and text -- and four diverse languages (English, German, Italian, and Chinese), enabling a comprehensive evaluation of MLLMs' abilities to interpret instructions across languages and combine them with multimodal contextual information.

License

• CC-BY-4.0

Dataset Sources

• Repository: MCIF
• Paper: MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks

Dataset Structure

Data Config

This dataset contains 4 splits organized by three dimensions following the split naming convention {track}_{prompt_type}.

Track - Input duration:

• long: Full-length, unsegmented inputs
• short: Pre-segmented inputs

Prompt Type - Prompt variation:

• fixed: Standardized prompts across all examples
• mixed: Includes prompt variations

Please note that all splits share the same set of original input audio and video files. The splits are meant to facilitate testing various use cases.

If you are interested in a specific task among those covered (Recognition, Translation, Question Answering, and Summarization), you can identify the relevant samples using the task field (respectively, ASR, TRANS, QA, and SUM) in each sample available from the references. The corresponding id field can then be used to match samples with their counterpart in this HuggingFace dataset.

Dataset Fields

The audio/video paths are relative within this repo.

You can download the data by cloning this repo:

git clone https://huggingface.co/datasets/FBK-MT/MCIF

And load the desired split by running:

dataset = load_dataset("FBK-MT/MCIF", "{long,short}_{fixed,mixed}prompt")

References

The references are available in MCIF.{short,long}.{en,de,it,zh}.ref.xml.gz (navigate to "Files and versions" tab or clone this repo).

IWSLT 2025 Version

Part of MCIF was used in the IWSLT 2025 instruction-following track.

This test data is available under branch IWSLT2025. You can access it by

dataset = load_dataset("FBK-MT/MCIF", "{en,de,it,zh}_{long,short}", revision="IWSLT2025")

Evaluation

Please use the official evaluation scripts from the MCIF GitHub Repo. The references are also available there.

## Changelog

Version 1.2

• Fixed summarization references

Version 1.1

• Fixed German summarization prompt
• Renamed files not to include version name in the filename

Citation

@inproceedings{papi2026mcif,
title={{MCIF}: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks},
author={Sara Papi and Maike Z{\"u}fle and Marco Gaido and Beatrice Savoldi and Danni Liu and Ioannis Douros and Luisa Bentivogli and Jan Niehues},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=PtPYZYfa0h}
}

Dataset Card Contact

@spapi and @danniliu