CALVIN-Para: A CALVIN Port of LIBERO-Para

CALVIN-Para is a port of LIBERO-Para to the CALVIN benchmark. It reuses the same two-axis paraphrase taxonomy, the same 43 type combinations, and the same PRIDE metric, applied to a different simulator, robot, and task family.

The purpose is cross-benchmark transferability: LIBERO-Para measures paraphrase robustness on LIBERO-Goal, and CALVIN-Para asks whether the same findings hold when only the benchmark changes.

All three LIBERO-Para findings reproduce on CALVIN. Paraphrase fragility persists regardless of model capability (Finding 1), object grounding remains the primary bottleneck (Finding 2), and failures stay planning-level rather than execution-level (Finding 3), even though the simulator, the robot, and the task family have all changed.

📄 Paper: arXiv:2603.28301 (Appendix E) 💻 Code: GitHub 🔗 Original benchmark: HAI-Lab/LIBERO-Para

Overview

LIBERO-Para is built on LIBERO-Goal, where all tasks share an identical initial state so that the instruction is the sole cue for task identification. CALVIN-Para reproduces that property on CALVIN: all 15 tasks live in a single scene (CALVIN scene D) where the drawer, sliding door, sliding cabinet, button, LED light, light bulb, and the red, blue, and pink blocks are present in every episode. Renaming an object therefore forces disambiguation against on-screen distractors.

As in LIBERO-Para, only the instruction is paraphrased. The visual scene, physics, and initial state are held fixed, and every paraphrase episode is paired to a canonical episode that shares its initial state.

Key Statistics

• 1,935 paraphrased instructions total
• 15 original CALVIN instructions
• 43 distinct paraphrase type combinations (Action × Object)
• 45 samples per variation type cell, uniformly

Benchmark Design

The taxonomy is unchanged from LIBERO-Para. See the original dataset card for the full definitions.

Object Axis (3 types), based on the Extended Paraphrase Typology (EPT; Kovatchev et al., 2018): Addition, SP-contextual, SP-habitual.

Action Axis (10 types): Lexical (Addition, SP-contextual, SP-habitual), Structural (Coordination, Subordination), Pragmatic (Need-statement, Embedded-imperative, Permission-directive, Question-directive, Hint).

Composing both axes yields 30 combined types, for 43 cells in total.

Dataset Statistics per Cell

Unlike LIBERO-Para, where verifier rejections left cells unevenly filled, every CALVIN-Para cell holds exactly 3 paraphrases per task across 15 tasks.

Original Instructions (from CALVIN)

Each of the 15 tasks carries 129 paraphrases (43 cells × 3).

CALVIN defines 34 tasks. Color variants of the same skill were dropped so that the object axis stays meaningful, and tasks requiring prior manipulation (place_in_slider, stack_block, and similar) were dropped because CALVIN generates initial states only for the first task of a rollout. See the GitHub README for the full derivation.

Usage

File Structure

CALVIN-Para/
├── README.md
├── LICENSE
├── paraphrases.json          # 1,935 paraphrases with taxonomy metadata
├── instructions.csv          # 15 canonical CALVIN instructions
└── episodes/
    ├── baseline_seed7.json   # 300 canonical episodes (15 tasks × 20 trials)
    └── paraphrase_seed7.json # 1,935 paraphrase episodes

Paraphrase Format

paraphrases.json holds the instructions and their taxonomy labels:

{
  "paraphrase_id": "p_00001",
  "base_task_id": "close_drawer",
  "base_original": "push the handle to close the drawer",
  "object_type": "ad",
  "action_type": "ad",
  "cell_id": "obj_ad__act_ad",
  "paraphrase_index": 1,
  "paraphrase": "carefully push the handle to close the storage drawer"
}

Episode Format

Where LIBERO-Para ships bddl files, CALVIN-Para ships episode definitions. Each entry pairs a CALVIN initial state with the instruction to evaluate:

{
  "episode_idx": 0,
  "init_state": {"drawer": "closed", "slider": "right", "lightbulb": 1,
                 "led": 0, "red_block": "table", "blue_block": "slider_right",
                 "pink_block": "table", "grasped": 0},
  "task_id": "open_drawer",
  "lang_override": "pull the handle to open the drawer",
  "metadata": {"kind": "baseline", "trial_idx": 0, "seed": 7}
}

baseline_seed7.json and paraphrase_seed7.json draw from the same initial-state pool, so each paraphrase episode has a canonical counterpart starting from the identical scene.

Running an Evaluation

The episode files are consumed by the CALVIN-Para evaluation pipeline, which resets calvin_env to init_state, injects lang_override, and logs per-step actions and end-effector positions:

GPU_ID=0 \
EPISODES_JSON=episodes/paraphrase_seed7.json \
EPISODES_OUTPUT_DIR=results/<model>/paraphrase_seed7 \
    bash paraphrase_eval/run_<model>_episodes.sh

For model setup guides, the per-episode eval patches, and analysis scripts, see the GitHub repository.

Citation

@misc{kim2026liberoparadiagnosticbenchmarkmetrics,
      title={LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models}, 
      author={Chanyoung Kim and Minwoo Kim and Minseok Kang and Hyunwoo Kim and Dahuin Jung},
      year={2026},
      eprint={2603.28301},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2603.28301}, 
}

Acknowledgments

• This project is built upon LIBERO by Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone.
• The cross-benchmark evaluation is built upon CALVIN by Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard.
• This research was supported by the AI Computing Infrastructure Enhancement (GPU Rental Support) User Support Program funded by the Ministry of Science and ICT (MSIT), Republic of Korea (RQT-25-090040).

License

This dataset is released under the MIT License.