MetricInstruct

The MetricInstrcut dataset consists of 44K quadruple in the form of (instruction, input, system output, error analysis) for 6 text generation tasks and 22 text generation datasets. The dataset is used to fine-tune TIGERScore, a Trained metric that follows Instruction Guidance to perform Explainable, and Reference-free evaluation over a wide spectrum of text generation tasks.

Project Page | Paper | Code | Demo | TIGERScore-7B | TIGERScore-13B

We present the MetricInstruct dataset, which is employed to fine-tune TIGERScore. The three underlying criteria for dataset construction are:

1. Dataset diversity: we choose 22 distinctive datasets as the source context to cover enough generation tasks.
2. Error coverage: we take system outputs generated from 50+ text generation systems to cover all types of errors and guarantee a balanced distribution.
3. Quality ensurance: to ensure MetricInstruct is tailored to gather in-depth error analysis, we sourced it by prompting OpenAI GPT models and then filtered through different heuristics to eliminate low-quality error analysis.

Data Source

Our system outputs come from two channels, namely real-world system outputs and synthetic outputs. The real-world system outputs are obtained from real systems, which ensures the error distribution is aligned with real-world ones. Check out our paper for more details.

Data Format

The dataset consists of 44K quadruple in the form of (instruction, input, system output, error analysis). For each item in the dataset, instruction is its task instruction, input_context is its input source, and hypo_output is the generated output, and errors is the error analysis given by ChatGPT or GPT-4.

Formatting

To format the data fields into a single prompt for finetuning or testing, We provide the following code for users to refer:

FINETUNE_INST = "You are evaluating errors in a model-generated output for a given instruction."
FINETUNE_INPUT = """\
Instruction: ${generation_instruction}
${input_context}


Model-generated Output:
${hypothesis_output}


For each error you give in the response, please also elaborate the following information:
- error location (the words that are wrong in the output)
- error aspect it belongs to.
- explanation why it's an error, and the correction suggestions.
- severity of the error ("Major" or "Minor").
- reduction of score (between 0.5 and 5 given the severity of the error)

Your evaluation output:
"""
inst_part = Template(FINETUNE_INST)
inst_part = inst_part.substitute()
input_part = Template(FINETUNE_INPUT)
input_part = input_part.substitute(
    generation_instruction=instruction,
    input_context=input_context,
    hypothesis_output=hypo_output
)
prompt = (inst_part + "\n" + input_part).strip("\n ") + "\n"
encodings = tigerscore_tokenizer(prompt, return_tensors="pt")
input_ids = encodings["input_ids"].to(tigerscore_model.device)
attention_mask = encodings["attention_mask"].to(tigerscore_model.device)

Example of formatted prompt:

You are evaluating errors in a model-generated output for a given instruction.
Instruction: Translate the following text from German to English.
Der künftige EM-Cheforganisator Philipp Lahm soll laut Grindel im DFB-Präsidium mitarbeiten.


Model-generated Output:
According to Grindel, the future head of the European Championships, Philipp Lahm, is to participate in the DFB Presidency.


For each error you give in the response, please also elaborate the following information:
- error location (the words that are wrong in the output)
- error aspect it belongs to.
- explanation why it's an error, and the correction suggestions.
- severity of the error ("Major" or "Minor").
- reduction of score (between 0.5 and 5 given the severity of the error)

Your evaluation output:

Citation

@article{jiang2023TIGERScore,
  title={TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks},
  author={Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, Wenhu Chen},
  journal={arXiv preprint arXiv:2310.00752},
  year={2023}
}