Structure Conditional MBR Decoding - Instruction Following Evaluation

Use Python 3.11 for this project.

Recommended usage: single GPU with at least 32GB RAM.

Overview

This repository evaluates instruction following capabilities of structure conditional MBR methods for LLMs on two benchmarks:

• AlpacaEval (tatsu-lab/alpaca_eval)
• MT-Bench (single-turn and multi-turn)

The main entrypoint main.py runs both datasets end-to-end and writes per-method scores to model_scores.txt.

Installation

Run the following command inside your virtual environment:

pip install -r requirements.txt

MT-Bench files preparation

Before running the evaluation on MT-Bench, you should prepare the data/mt_bench/question.jsonl and data/mt_bench/gpt-4o-full.jsonl files.

• Download MT-Bench questions from the original dataset's repository and save them as question.jsonl. The format of each line should be as follows: {"question_id": 1, "category": "some_category_name", "turns": ["turn_1", "turn_2"]}.

• Collect reference answers for those answers and save them as gpt-4o-full.jsonl. The format of each line should be as follows: {"question_id": 1, "answer_id": "1", "model_id": "your_model_id", "choices": [{"index": 0, "turns": ["turn_1", "turn_2"]}], "tstamp": some_timestamp} You can either use the uploaded reference answers on MT-Bench official repository, or use our script generate_openai.py (instructions on running that can be found below) to generate reference answers yourself.

Project structure

• main.py — Main file. Loads a generation model and a Prometheus judge, runs AlpacaEval and MT-Bench single/multi-turn, and saves method scores.

  • Argument --template chooses the prompting format: INST or CHAT-TEMPLATE (default).
  • Writes model_scores.txt with per-method Prometheus win rates.

• generate_openai.py — Generate MT-Bench-style JSONL answers with OpenAI’s API.

  • Set API_KEY, DATASET_PATH, MODEL_NAME at the top.
  • Class OpenAIModel wraps chat completions; MTBenchModel formats MT-Bench turns and writes answers.jsonl compatible with MT-Bench consumer tools.

• mbr.py — Core selection algorithms:

  • standard_mbr(utility_matrix) — Classic MBR using mean utility.
  • cutoff_mbr_abs(...) and cutoff_mbr_dev(...) — Apply cutoffs before averaging utilities.
  • cluster_mbr(...) — Apply cluster embeddings (Sentence-Transformers) and pick the best within the dominant cluster; can also return a full merged ranking.
  • cosine_mbr(...) — Weight utility by cosine similarity of sentence embeddings.

• generate_methods/

  • generate_candidates_greedy.py — Greedy decoding for prompts; returns generated texts.
  • generate_candidates_mbr.py — Samples num_candidates candidates with temperature sampling, computes BERTScore F1 utilities (pairwise), and returns the MBR-best candidate plus utilities.

• datasets_eval/

  • alpaca_eval.py — Evaluation pipeline for AlpacaEval.
  • mt_bench.py — Evaluation pipeline for MT-Bench.
  • prometheus_judge.py — Wrapper around PrometheusEval. Uses a rubric to compute relative grades vs. reference answers and returns count of wins.

How to run

There are two typical workflows: generate+evaluate or evaluate-only (using existing caches).

1) Generate candidates and evaluate (fresh run)

Edit main.py to enable a generation model:

1. Uncomment the model/tokenizer lines near the top and set your model name, e.g.:
   • model_name = "allenai/OLMo-2-1124-13B-Instruct"
   • model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16).eval().to(device)
   • tokenizer = AutoTokenizer.from_pretrained(model_name); tokenizer.padding_side = "left"
2. Comment out the temporary placeholders that set model = None and tokenizer = None.
3. Ensure data/mt_bench/question.jsonl and data/mt_bench/gpt-4o-full.jsonl exist.
4. Make sure the cache/ directory exists.

Run:

python main.py --template CHAT-TEMPLATE

2) Evaluate-only using cached outputs

If you already have the pickled caches, you can:

1. Leave the generation model commented out and keep:
   • model = None and tokenizer = None (these aren’t used when loading caches).
2. Ensure the relevant cache files exist under cache/.

Run:

python main.py --template CHAT-TEMPLATE

The scripts will load caches and only run Prometheus judging, producing model_scores.txt.

MT-Bench: generating reference answers via OpenAI

If you would like to use OpenAI chat models to produce MT-Bench reference answers in JSONL format:

1. Open generate_openai.py and set:
   • API_KEY to your OpenAI API key
   • DATASET_PATH if your MT-Bench question file location differs
   • MODEL_NAME to your desired model (e.g., gpt-4.1-mini, gpt-4o-mini)
2. Run the script:

python generate_openai.py