LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models

LIBERO-Para is a controlled benchmark for evaluating the paraphrase robustness of Vision-Language-Action (VLA) models. It independently varies action expressions and object references—the two core linguistic components of robotic manipulation instructions—enabling fine-grained analysis of how different types of linguistic variation affect VLA performance.

📄 Paper: arXiv:2603.28301 💻 Code: GitHub

Overview

LIBERO-Para is constructed on top of LIBERO-Goal, where all tasks share an identical initial state, making the instruction the sole cue for task identification. The benchmark paraphrases only the instructions while keeping all other factors (visual scene, physics, etc.) fixed. All paraphrases are held out for evaluation only.

Key Statistics

• 4,092 paraphrased instructions total
• 10 original LIBERO-Goal instructions
• 43 distinct paraphrase type combinations (Action × Object)
• ~100 samples per variation type cell

Benchmark Design

Two-Axis Paraphrase Scheme

LIBERO-Para adopts a two-axis design grounded in established linguistic taxonomies:

Object Axis (3 types) — Lexical variation of object references, based on the Extended Paraphrase Typology (EPT; Kovatchev et al., 2018):

• Addition: Adding functional descriptors (e.g., "stove" → "gas stove")
• SP-contextual: Contextually appropriate substitution (e.g., "stove" → "cooktop")
• SP-habitual: Common synonym substitution (e.g., "stove" → "cooker")

Action Axis (10 types) — Variation in how actions are linguistically expressed:

Compositional Variation

Beyond individual axes, the benchmark includes compositional paraphrases that vary both action and object expressions simultaneously, yielding 30 combined types (3 Object × 10 Action).

Dataset Statistics per Cell

Original Instructions (from LIBERO-Goal)

Usage

File Structure

LIBERO-Para/
├── README.md
└── bddl_files/
    └── ...

Using with LIBERO

The bddl files can be used directly with the LIBERO evaluation environment. Place the downloaded bddl files into the appropriate LIBERO directory:

mv bddl_files/* /path/to/libero/libero/bddl_files/

For detailed evaluation guides, model-specific setup instructions, and analysis scripts, please refer to our GitHub repository.

Citation

@misc{kim2026liberoparadiagnosticbenchmarkmetrics,
      title={LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models}, 
      author={Chanyoung Kim and Minwoo Kim and Minseok Kang and Hyunwoo Kim and Dahuin Jung},
      year={2026},
      eprint={2603.28301},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2603.28301}, 
}

Acknowledgments

• This project is built upon LIBERO by Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone.
• This research was supported by the AI Computing Infrastructure Enhancement (GPU Rental Support) User Support Program funded by the Ministry of Science and ICT (MSIT), Republic of Korea (RQT-25-090040).

License

This dataset is released under the MIT License.