The Power of Context: An In-depth Analysis of Large Language Models and Their Evaluation Using Custom-Tailored Benchmark Sets

This work examines the impact of varying context lengths on the output of large language models(LLMs) and explores the performance of these models after training with additional information.

File Introduction

• Benchmark

  This folder contains the four evaluation metrics for language models: BLEU, ROUGE, Perplexity, and WMD, along with their summaries.

  • BLEU: Measures the precision of the text generated by the LLMs.

  • ROUGE: Measures the recall of the text generated by the LLMs.

  • Perplexity: Evaluates the predictive ability of the LLMs.

  • WMD: Assesses the semantic similarity between the text generated by the LLMs and the reference text.

• Datenbank

  Three datasets are used in this work, namely ProofWriter, 2WikiMultihopQA and HotpotQA. (Refer to: Related Links)

  The id and the required context from these three databases are stored in a MySQL table for creating contexts.

• Datenset

  Extract the questions and answers from the three datasets to train the LLMs.

• Diagramm

  Plot box plots for each model with different context lengths and plot the loss function during the training process.

• Kontexterstellung

  Create contexts of different lengths as needed.

• Eingabe

  The code in this folder allows you to input queries into the five large language models as well as the trained models.

• Training

  Use the LoRA method and the data generated in the Datenset file to train the LLMs.

Related Links

• ProofWriter

  https://allenai.org/data/proofwriter

• 2WikiMultihopQA

  https://github.com/Alab-NII/2wikimultihop?tab=readme-ov-file

• HotpotQA

  https://hotpotqa.github.io/

• Dataset used for training this work

  https://github.com/Yitong-Jing/Analysis-LLMs/tree/main/Datenset/datenset

• Database used to create context for this work

  https://drive.google.com/file/d/15pc3iV9juJS7uLqsGThivWDiVy2QeBsA/view?usp=drive_link