Project Links

Dataset Description

The IMPACT dataset contains 50 human created prompts for each category, 200 in total, to test LLMs general writing ability.

Instructed LLMs demonstrate promising ability in writing-based tasks, such as composing letters or ethical debates. This dataset consists prompts across 4 diverse usage scenarios:

• Informative Writing: User queries such as self-help advice or explanations for various concept
• Professional Writing: Format involves suggestions presentations or emails in a business setting
• Argumentative Writing: Debate positions on ethical and societal question
• Creative Writing: Diverse writing formats such as stories, poems, and songs.

The IMPACT dataset is included in our InstructEval Benchmark Suite.

Evaluation Results

We leverage ChatGPT to judge the quality of the generated answers by LLMs. In terms of:

• Relevance: how well the answer engages with the given prompt
• Coherence: general text quality such as organization and logical flow

Each answer is scored on a Likert scale from 1 to 5. We evaluate the models in the zero-shot setting based on the given prompt and perform sampling-based decoding with a temperature of 1.0

Citation

Please consider citing the following article if you found our work useful:

bibtex
@article{chia2023instructeval,
      title={INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models}, 
      author={Yew Ken Chia and Pengfei Hong and Lidong Bing and Soujanya Poria},
      journal={arXiv preprint arXiv:2306.04757},
      year={2023}
}