[Paper] [HF Collection] [Code]

Authors: Yapei Chang, Yekyung Kim, Michael Krumdick, Amir Zadeh, Chuan Li, Chris Tanner, Mohit Iyyer

Contact: yapeic@umd.edu

TLDR > We extend RLVR beyond easily verifiable domains like math and code to the more open-ended setting of general instruction following. Surprisingly, we find that BLEU—a simple n-gram matching metric—when paired with high-quality references from strong LLMs, achieves human agreement comparable to 8B and 27B reward models on Chatbot Arena outputs. Based on this insight, we introduce BLEUBERI, which uses BLEU directly as a reward in GRPO training. BLEUBERI matches the performance of RM-guided GRPO across four instruction-following benchmarks and produces more factually grounded outputs, with human raters rating them on par with those from reward model-trained systems.

Dataset Creation

Creating the data pool

This dataset is derived from the Tulu3 SFT mixture, which contains 939K instruction-following examples sourced from 18 diverse datasets. To construct the 50K subset used in BLEUBERI training, we followed a multi-step filtering and sampling process:

1. Domain Focus: Among all 18 data sources in Tulu3, we focus on the five instruction-following data sources within this mixture: FLAN v2, No Robots, OpenAssistant Guanaco, Tulu 3 Persona IF, and WildChat GPT-4.

2. Sampling:
   From these, we filter out examples with instructions or responses shorter than 10 tokens or greater than 512 tokens (based on Qwen2.5-7B tokenizer), and those that are not in English (labaled by langdetect). Then, we sample 50K examples, trying to balance between the 5 instruction-following data sources. See the image below for distribution across data sources.

Distribution of data sources among our 50K pool.

Collecting synthetic references

While Tulu3 already provides responses for each instruction, we additionally collect synthetic references from 5 powerful LLMs: claude-3-7-sonnet@20250219, deepseek-chat-v3, gemini-2.5-pro-exp-03-25, o4-mini-2025-04-16, Llama-3.1-8B-Instruct. These synthetic references are included as part of this BLEUBERI-Tulu3-50k dataset.

Use in BLEUBERI

The 50K dataset served as a candidate pool from which the 5,000 lowest-BLEU-scoring prompts were selected for GRPO training with BLEU-based rewards.