🐍 LongICLBench

This repo contains the official code and data for our TMLR25 paper Long-context LLMs Struggle with Long In-context Learning

🤗 HuggingFace Dataset

🌟 Leaderboard

We created LongICLBench to conduct comprehensive evaluations of Large Language Models (LLMs) on extreme-label classification challenges with in-context learning. We compiled six datasets that encompass a broad spectrum of labels, ranging from 28 to 174 categories, and varied the lengths of input (from few-shot demonstrations) between 2K and 50K tokens to ensure thorough testing Our principal discoveries reveal:

• 1️⃣ A notable decline in the performance of even the most advanced models as the complexity of the tasks increases.
• 2️⃣ In the most challenging scenarios, especially those with 174 classes, models find it difficult to discern subtle differences.
• 3️⃣ The performance of certain models is sensitive to the position of the instances in the demonstrations.

🛠️ Set up

git clone https://github.com/TIGER-AI-Lab/LongICLBench.git
cd LongICLBench
pip install -r requirements.txt

🌈 Benchmarking

We collect six extreme-label classification datasets with details as below:

🎖️ Model Evaluation

We evaluate the performance of 13 long-context large language models. Their information are presented as follows:

The evaluation results for some of the datasets is presented as follows: (Please check more details in our paper)

🕹️ Run the script to re-obtain the demonstrated results: (You can just set [api_key] to '' for the locally deployed models)

sh script_goemotion.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]
sh script_bank77.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]
sh script_fewnerd.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]
sh script_tacred.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]
sh script_dialogue.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]
sh script_discovery.sh [model_short_name] [CUDA_DEVICES] 500 [api_key]

To evaluate the results for Few-NERD and DialogRE with F1 score:

python fewnerd_evaluate.py --model [model_short_name] --round [number_of_rounds]
python dialogueRE_evaluate.py --model [model_short_name] --round [number_of_rounds]

🖊️ Citation

Please kindly cite our paper if you find our study useful:

@misc{li2024longcontext,
      title={Long-context LLMs Struggle with Long In-context Learning}, 
      author={Tianle Li and Ge Zhang and Quy Duc Do and Xiang Yue and Wenhu Chen},
      year={2024},
      eprint={2404.02060},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}