Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions

CanItEdit is a benchmark for evaluating LLMs on instructional code editing, the task of updating a program given a natural language instruction. The benchmark contains 105 hand-crafted Python programs with before and after code blocks, two types of natural language instructions (descriptive and lazy), and a hidden test suite.

The dataset’s dual natural language instructions test model efficiency in two scenarios:

1. Descriptive: Detailed instructions replicate situations where users provide specific specifications or another model outlines a plan, similar to Reflexion prompting,
2. Lazy: Informal instructions resemble typical user queries for LLMs in code generation.

For more information and results see our paper.

Citation

If you use our work, please cite our paper as such:

@inproceedings{cassano2023edit,
      title={{Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions}}, 
      author={Federico Cassano and Luisa Li and Akul Sethi and Noah Shinn and Abby Brennan-Jones and Anton Lozhkov and Carolyn Jane Anderson and Arjun Guha},
      booktitle={The First International Workshop on Large Language Model for Code},
      year={2024},
      url={https://arxiv.org/abs/2312.12450}
}

How To Evaluate

All the code for evaluating the benchmark can be found in our GitHub repository.