CDR-Bench

CDR-Bench evaluates whether language models can faithfully execute compositional, order-sensitive data refinement recipes. A model receives raw text and a natural-language recipe, then returns the refined text and the final KEEP / DROP decision.

Links:

• Code: https://github.com/lukahhcm/data-juicer-hub/tree/CDR-Bench
• Dataset: https://huggingface.co/datasets/lukahh/CDR-Bench
• Paper: https://arxiv.org/abs/2606.31435

Splits and Configs

The Hugging Face viewer exposes two configs, each with a single test split:

The main config is the five-track paper benchmark. The semantic_extension config contains appendix-style real-scenario extensions adapted from external benchmarks; these are evaluated as atomic-vs-compositional comparisons and are not merged into the main paper tracks.

The hallucination extension is balanced at the base-sample level: it contains 300 FAVA-derived base samples, with 150 hallucinated and 150 clean examples. The atomic hallucination track expands those samples into four subtasks per sample, yielding 1,200 atomic rows; the compositional hallucination track has 300 rows.

The additional semantic domains are fixed at 300 base samples each. HelpSteer2 rubric scoring expands into five atomic score dimensions plus one compositional all-dimensions task. Aegis 2.0 safety tagging is balanced across observed prompt/response label pairs and expands into three atomic tag fields plus one compositional all-fields task.

Data Sources and Attribution

CDR-Bench is built upon or references several open-source and research datasets. We thank the original authors and dataset maintainers for making these resources available. The table below summarizes the upstream sources used or referenced in the benchmark construction pipeline.

The original datasets remain the property of their respective owners. CDR-Bench adds additional processing, including operator mining over unordered operator sets, composition recipe alignment, prompt materialization, and benchmark packaging for evaluating LLM data-refinement capabilities. Users should consult and comply with the terms of the upstream datasets when using CDR-Bench, especially for non-academic, commercial, or redistribution use cases.

License

The CDR-Bench code is released under the Apache 2.0 License.

The CDR-Bench dataset is released for non-commercial academic research use. Because the benchmark is constructed from multiple upstream datasets, users must also comply with the original licenses and terms of the corresponding source datasets. In cases where an upstream dataset imposes stricter terms, those terms take precedence for the derived subset.

Repository Layout

data/
  main/
    test.jsonl
  semantic_extension/
    test.jsonl
manifest.json
benchmark_v3_all.jsonl
benchmark_v3_all_prompts.jsonl
tracks/
  # default evaluation files; main tracks are materialized to paper seed=0, K=3
  atomic_m.jsonl
  atomic_f.jsonl
  agnostic_m.jsonl
  order_m.jsonl
  order_f.jsonl
  semantic_pii_atomic.jsonl
  semantic_pii_compositional.jsonl
  semantic_hallu_atomic.jsonl
  semantic_hallu_compositional.jsonl
  semantic_rubric_atomic.jsonl
  semantic_rubric_compositional.jsonl
  semantic_safety_atomic.jsonl
  semantic_safety_compositional.jsonl
tracks_all_prompts/
  # full main-track prompt pools for custom prompt-seed sweeps
  atomic_m.jsonl
  atomic_f.jsonl
  agnostic_m.jsonl
  order_m.jsonl
  order_f.jsonl

Use data/main/test.jsonl and data/semantic_extension/test.jsonl for standard Hugging Face loading. Use tracks/*.jsonl when running the release scripts, which evaluate each track separately. For the five main tracks, tracks/ is the paper-aligned default: each row already contains the three prompt variants selected with seed 0 for RS@3. Use tracks_all_prompts/ only when explicitly running custom prompt-seed sweeps from the full prompt pools.

In the Hugging Face config files, pii_meta, hallu_meta, rubric_meta, and safety_meta are stored as JSON strings for stable viewer schema inference. The per-track files preserve those metadata fields as JSON objects.

Schema

Each row contains the input, reference output, operator metadata, prompt variants, and scoring contract. Important fields include:

Metrics

Recipe Success (RS) is reported for all rows. Refinement Gain (RG) is reported only for text-output rows where reports_refinement_gain=true. JSON-only semantic subtasks use canonical JSON exact match as RS.

Hallucination compositional rows use mixed_structured_text: RS is computed from exact canonical JSON matching over detection, span extraction, type classification, and corrected_text, while the corrected_text field is also available for text-refinement diagnostics.

Citation

If you use CDR-Bench, please cite the paper:

@misc{huang2026cdrbenchevaluatingfaithfulexecution,
      title={CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipes}, 
      author={Yuchen Huang and Xiang Li and Zhenqing Ling and Sijia Li and Qianli Shen and Daoyuan Chen and Yi R. Fung and Yaliang Li},
      year={2026},
      eprint={2606.31435},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2606.31435}, 
}