This dataset contains 21 many-shot ICL tasks that are designed to evaluate the long-context capability of LLMs, as introduced in the paper On Many-Shot In-Context Learning for Long-Context Evaluation. We categorize the tasks into similar-sample learning (SSL) and all-sample learning (ASL) groups.

SSL Tasks: banking77, dialogRE, TREC50, CLINC150, and BBH_geometric_shapes

ASL Tasks: GSM8K, MATH-algebra, MATH-counting_and_probability, MATH-geometry, MATH-number_theory, XLSUM, GPQA_cot, ARC_challenge, BBH-dyck_languages, BBH-salient_translation_error_detection, and BBH-word_sorting.

SSL tasks predominantly assess models' skills to learn from similar examples rather than all examples. ASL tasks evaluate whether models assimilate and learn from all demonstrations. More discussions on these two tasks can be found in the paper.

Use this Python file to download the data.

Paper:https://arxiv.org/abs/2411.07130

Code:https://github.com/launchnlp/ManyICLBench

Leaderboard:https://huggingface.co/spaces/launch/ManyICLBench_Leaderboard

Citation Information

If you use this dataset, please cite the original paper:

@article{zou2025manyshotincontextlearninglongcontext,
      title={On Many-Shot In-Context Learning for Long-Context Evaluation}, 
      author={Kaijian Zou and Muhammad Khalifa and Lu Wang},
      year={2025},
      eprint={2411.07130},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.07130}, 
}