Pablo Valle1, Sergio Segura2, Shaukat Ali3, Aitor Arrieta1
Mondragon Unibertsitatea1, University of Seville2, Simula Research Laboratory3
In this paper, we explore whether Metamorphic Testing (MT) can alleviate the test oracle problem in this context. To do so, we propose two metamorphic relation patterns and five metamorphic relations to assess whether changes to the test inputs impact the original trajectory of the VLA-enabled robots. An empirical study involving five VLA models, two simulated robots, and four robotic tasks shows that MT can effectively alleviate the test oracle problem by automatically detecting diverse types of failures, including, but not limited to, uncompleted tasks. More importantly, the proposed MRs are generalizable, making the proposed approach applicable across different VLA models, robots, and tasks, even in the absence of test oracles.
The repository is organized as follows:
.sh) for Conda environments per VLA model.We provide a comprehensive benchmark for evaluating Vision-Language-Action (VLA) models, including the environments and simulators required to run experiments. Specifically, the benchmark supports five VLA models, Isaac-Groot, pi0, EO-1, OpenVLA, and SpatialVLA, across four manipulation tasks and two robotic platforms, corresponding to the Bridge and Fractal datasets.
For each task (pick up, move near, put in, and put on), we include 500 distinct scenes, along with the execution results for each VLA model, enabling standardized comparisons.
Additionally, the benchmark contains 9,320 follow-up test cases generated using the five proposed MRs. These follow-up cases were derived from the source test cases that successfully passed across all models and tasks. Complete execution results for these follow-up test cases are also provided, allowing thorough evaluation of both task-level correctness and execution-level robustness.
The software requirements to run this are minimal:
Note: "Minimal" applies to users who choose to run the provided Docker image. Running directly on your local machine requires additional software dependencies, which are detailed in Building from source.
The hardware requirements to run this are VLA dependant since each VLA needs a specific amount of GPU RAM, overall the following requirements are needed:
There are two ways to install and use this repository:
Docker simplifies the installation of robotics simulators and CUDA requirements. We provide a Dockerfile and a prebuilt image on Dockerhub. Ensure Docker is installed and GPU passthrough is enabled.
Build the image:
Download only the Docker file and build the image:
docker build -t mt_4_vlas .
Run the container with GPU support:
If you built from image:
docker run -d --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all --name="mt_4_vlas" mt_4_vlas
If using the prebuilt image:
docker run -d --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all --name="mt_4_vlas" pvalleentrena/mt_of_vlas
Enter the container and verify GPU access:
Inside the container, run:
docker exec -it mt_4_vlas bash
nvidia-smi
You should see something similar to this showing which is the status of the GPU associated to the Docker:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08 Driver Version: 570.172.08 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A6000 Off | 00000000:01:00.0 Off | Off |
| 30% 39C P0 73W / 300W | 0MiB / 49140MiB | 2% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
If you prefer a local installation, ensure CUDA 12.1+ is installed (tested on Ubuntu 22.04 and CUDA 12.1). Then install the following:
sudo apt-get update && sudo apt-get install -y \
dbus-x11 git locales pavucontrol pulseaudio pulseaudio-utils \
software-properties-common sudo vim x11-xserver-utils \
xfce4 xfce4-goodies xorgxrdp xrdp ffmpeg libaio-dev \
python3-pip python3-venv python3-dev \
vulkan-tools libvulkan1 mesa-vulkan-drivers libgl1-mesa-glx \
gnupg wget nano git-lfs xvfb ca-certificates
Note: Ensure NVIDIA drivers (recommended 525+) are installed.
sudo apt-get install -y libglvnd-dev
# if it doesn't exist
sudo mkdir -p /usr/share/vulkan/icd.d/ /usr/share/glvnd/egl_vendor.d/
# Configure Vulkan ICD if it doesn't exist
echo '{ "file_format_version" : "1.0.0", "ICD": { "library_path": "libGLX_nvidia.so.0", "api_version" : "1.2.155" } }' | sudo tee /usr/share/vulkan/icd.d/nvidia_icd.json
# Configure EGL Vendor if it doesn't exist
echo '{ "file_format_version" : "1.0.0", "ICD" : { "library_path" : "libEGL_nvidia.so.0" } }' | sudo tee /usr/share/glvnd/egl_vendor.d/10_nvidia.json
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
bash ~/miniconda.sh -b -p $HOME/miniconda
source "$HOME/miniconda/bin/activate"
conda init bash
git clone https://github.com/pablovalle/MT_of_VLAs.git
cd MT_of_VLAs
The following steps apply to both Docker and source installations.
Once everything is set up and you can access to the repository either on your local machine or inside the docker, for each VLA one conda environment will be generated and the corresponding models will be downloaded. To do so, inside environment_mount you can find one .sh file for each model. To setup the environment and download the models:
cd {this_repo/environment_mount}
./{model_name}.sh #Options: EO1, GR00T, PI0, SPATIALVLA, and OPENVLA
Note: Setting up the environment may take approximately 10–15 minutes, depending on your internet connection (Overall the fastest model in installation and execution is EO1). The model name corresponding to each conda environment is the same name but in lowercase:
eo1,gr00t,pi0and for spatialVLA and openVLA the number of parameters should be added:spatialvla-4bandopenvla-7b.
Once it finishes, you will find the model weights inside checkpoints folder and you will have the corresponding conda environment wiht the same name as the .sh file you launched. For example if you launched EO1.sh you will have a conda env called EO1.
To generate the follow-up test cases just a .sh file should be ran:
cd {this_repo/experiments}
./{follow_up_generator.sh}.sh -e <env> -m <model> [options]
Note: Pre-generated follow-up cases used in our evaluation are available in data/FollowUp. Remove, rename or move this folder if you want to regenerate test cases.
| Flag | Status | Description |
|---|---|---|
| -e | REQUIRED | Conda Environment: The name of the specific Conda environment you wish to activate for the run. |
| -m | REQUIRED | Model Name: The identifier for the model being tested (e.g., eo1, gr00t). |
| -r | OPTIONAL | Metamorphic Relations: Specifies which relations to apply (MR1 through MR5). Multiple values should be comma-separated. |
| -t | OPTIONAL | Task ID Filter: Filters the execution to specific tasks. Accepts arrays like [1,2,3] or ranges like [1-50]. |
| -d | OPTIONAL | Dataset JSON: Allows you to target specific dataset files (e.g., t-grasp_n-1000_o-m3_s-2498586606.json). |
| -h | OPTIONAL | Help: Displays the manual and all available options, then exits the script. |
Note: If no task ID is specified, to follow our experimental setup it will take all the passing test from the source test cases, so if that the case, please first run the source test cases and provide the human verification. In our case we took this data from Evaluating Uncertainty and Quality of Visual Language Action-enabled Robots.
Usage example:
./follow_up_generator.sh -e EO1 -m eo1 -r MR1,MR3 -t [1-10,15,18] -d t-grasp_n-1000_o-m3_s-2498586606.json
To execute the source test cases just a .sh file should be ran:
cd {this_repo/experiments}
./{run_source_test_cases.sh}.sh -e <env> -m <model> [options]
Note: Precomputed results for the source test cases are available in results. By default, the script will not re-run these test cases. To execute them again, you can rename, move, or remove the
/resultsfolder. Do not remove the/results/human_evalfolder
| Flag | Status | Description |
|---|---|---|
| -e | REQUIRED | Conda environment name to activate. |
| -m | REQUIRED | Specific model name to run (e.g., eo1, gr00t3). |
| -d | OPTIONAL | Dataset JSON filename(s). Accepts one or multiple values separated by commas (e.g., t-grasp_n-1000_o-m3_s-2498586606.json,t-move_n-1000_o-m3_s-2263834374.json). Default: Runs the four standard datasets (t-grasp_n-1000_o-m3_s-2498586606.json, t-move_n-1000_o-m3_s-2263834374.json, t-put-in_n-1000_o-m3_s-2905191776.json, t-put-on_n-1000_o-m3_s-2593734741.json). |
| -h | OPTIONAL | Displays the help message and exits. |
Usage example:
./run_source_test_cases.sh -e EO1 -m eo1 -d t-put-on_n-1000_o-m3_s-2593734741.json
To execute the follow-up test cases just a .sh file should be ran:
cd {this_repo/experiments}
./{run_follow_up_test_cases.sh}.sh -e <env> -m <model> [options]
Note: Precomputed results for the follow-up test cases are available in FollowUp_Results.By default, the script will not re-run these follow-up test cases. To execute them again, you can rename, move, or remove the
/FollowUp_Resultsfolder.
| Flag | Status | Description |
|---|---|---|
| -e | REQUIRED | Conda environment name to activate. |
| -m | REQUIRED | Specific model name to run (e.g., eo1, gr00t). |
| -r | OPTIONAL | Metamorphic Relations (MRs). Accepts single or multiple values separated by commas or spaces (e.g., "MR1,MR2,MR5"). Default: MR1, MR2, MR3, MR4, MR5. |
| -d | OPTIONAL | Dataset JSON filename(s). Accepts one or multiple values separated by commas (e.g., grasp,move). Default: Runs the four standard datasets (grasp, move, put-in, put-on). |
| -h | OPTIONAL | Displays the help message and exits. |
Usage example:
./run_source_test_cases.sh -e EO1 -m eo1 -r MR1,MR3 -d grasp,move
To reproduce the results reported in our paper, we provide precomputed outcomes for both the source test cases (results) and the follow-up test cases(FollowUp_Results), along with an automated analysis pipeline located in the result_analysis folder. The steps below describe how to analyze the provided results.
Note: The analysis scripts are preconfigured to generate results exactly as presented in the paper. If additional metamorphic relations (MRs) are added, the scripts must be updated to handle them correctly. Similarly, if any VLA models are added or removed, figure sizes may need adjustment.
Before running the analysis, ensure that all required results are present in their corresponding folders (this is in case you renamed, moved or removed any folder). Once ready, execute the following commands:
cd {this_repo/result_analysis}
./{result_analysis.sh}.sh -e <env>
Note: Specifying the environment is mandatory, as all required analysis libraries are included within each environment. You can perform the analysis using any of the provided environments.
Running the analysis will process the results for RQ1 and RQ2, generating a set of summary files named RQ1_results_{model_name}.xlsx, one per VLA model. Additionally, the 3rd and 4th figures from the paper, along with other relevant visualizations, will be produced in figures. The analysis will also generate the folders output_mr and output_oracle, containing the selected videos used to build the taxonomy described in RQ3.
The taxonomy itself was generated using a questionnaire, which is available here: questionnaire
RuntimeError: vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed
Some required Vulkan extension is not present. You may not use the renderer to render, however, CPU resources will be still available.
Segmentation fault (core dumped)
Follow this link to troubleshoot the issue. (Even though the doc points to SAPIEN 3 and ManiSkill3, the troubleshooting section still applies to the current environments that use SAPIEN 2.2 and ManiSkill2).
TypeError: 'NoneType' object is not subscriptable
Please also refer to the original repo or vulkan_setup if you encounter any problems.
tensorflow-2.15.0 conflicts with tensorflow-2.15.1?
The dlimp library has not been maintained for a long time, so the TensorFlow version might be out of date. A reliable solution is to comment out tensorflow==2.15.0 in the requirements file, install all other dependencies, and then install tensorflow==2.15.0 finally. Currently, using tensorflow==2.15.0 has not caused any problems.
If you find this project useful in your research, please consider cite:
@inproceedings{valle2026mt_of_vlas_icst,
title = {Metamorphic Testing of Vision-Language Action-Enabled Robots},
author = {Pablo Valle and Sergio Segura and Shaukat Ali and Aitor Arrieta},
booktitle = {Proceedings of the 19th IEEE International Conference on Software Testing, Verification and Validation (ICST 2026)},
year = {2026},
month = may,
address = {Daejeon, Republic of Korea},
organization = {IEEE},
note = {\url{https://arxiv.org/abs/2602.22579}},
url = {https://arxiv.org/abs/2602.22579}
}
This project is built with reference to the code of the following projects: Isaac-Groot, Lerobot, EO-1, SimplerEnv-OpenVLA, VLATest, and Evaluating Uncertainty and Quality of Visual Language Action-enabled Robots.
Pablo Valle and Aitor Arrieta are part of the Software and Systems Engineering research group of Mondragon Unibertsitatea (IT1519-22), supported by the Department of Education, Universities and Research of the Basque Country. Pablo Valle is supported by the Pre-doctoral Program for the Formation of Non-Doctoral Research Staff of the Education Department of the Basque Government (Grant n. PRE_2025_2_0252). Shaukat Ali is supported by the Co-tester project (No. 314544) funded by the Research Council of Norway. This work has been partially funded by the Spanish Ministry of Science, Innovation and Universities (project PID2023-152979OA-I00), funded by MCIU/AEI/10.13039/501100011033/FEDER, UE. This paper is also part of the project PID2024-156482NB-I00, funded by MICIU/AEI/10.13039/501100011033 and by the ESF+.
Pablo Valle1, Sergio Segura2, Shaukat Ali3, Aitor Arrieta1
Mondragon Unibertsitatea1, University of Seville2, Simula Research Laboratory3
In this paper, we explore whether Metamorphic Testing (MT) can alleviate the test oracle problem in this context. To do so, we propose two metamorphic relation patterns and five metamorphic relations to assess whether changes to the test inputs impact the original trajectory of the VLA-enabled robots. An empirical study involving five VLA models, two simulated robots, and four robotic tasks shows that MT can effectively alleviate the test oracle problem by automatically detecting diverse types of failures, including, but not limited to, uncompleted tasks. More importantly, the proposed MRs are generalizable, making the proposed approach applicable across different VLA models, robots, and tasks, even in the absence of test oracles.
The repository is organized as follows:
.sh) for Conda environments per VLA model.We provide a comprehensive benchmark for evaluating Vision-Language-Action (VLA) models, including the environments and simulators required to run experiments. Specifically, the benchmark supports five VLA models, Isaac-Groot, pi0, EO-1, OpenVLA, and SpatialVLA, across four manipulation tasks and two robotic platforms, corresponding to the Bridge and Fractal datasets.
For each task (pick up, move near, put in, and put on), we include 500 distinct scenes, along with the execution results for each VLA model, enabling standardized comparisons.
Additionally, the benchmark contains 9,320 follow-up test cases generated using the five proposed MRs. These follow-up cases were derived from the source test cases that successfully passed across all models and tasks. Complete execution results for these follow-up test cases are also provided, allowing thorough evaluation of both task-level correctness and execution-level robustness.
The software requirements to run this are minimal:
Note: "Minimal" applies to users who choose to run the provided Docker image. Running directly on your local machine requires additional software dependencies, which are detailed in Building from source.
The hardware requirements to run this are VLA dependant since each VLA needs a specific amount of GPU RAM, overall the following requirements are needed:
There are two ways to install and use this repository:
Docker simplifies the installation of robotics simulators and CUDA requirements. We provide a Dockerfile and a prebuilt image on Dockerhub. Ensure Docker is installed and GPU passthrough is enabled.
Build the image:
Download only the Docker file and build the image:
docker build -t mt_4_vlas .
Run the container with GPU support:
If you built from image:
docker run -d --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all --name="mt_4_vlas" mt_4_vlas
If using the prebuilt image:
docker run -d --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all --name="mt_4_vlas" pvalleentrena/mt_of_vlas
Enter the container and verify GPU access:
Inside the container, run:
docker exec -it mt_4_vlas bash
nvidia-smi
You should see something similar to this showing which is the status of the GPU associated to the Docker:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08 Driver Version: 570.172.08 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A6000 Off | 00000000:01:00.0 Off | Off |
| 30% 39C P0 73W / 300W | 0MiB / 49140MiB | 2% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
If you prefer a local installation, ensure CUDA 12.1+ is installed (tested on Ubuntu 22.04 and CUDA 12.1). Then install the following:
sudo apt-get update && sudo apt-get install -y \
dbus-x11 git locales pavucontrol pulseaudio pulseaudio-utils \
software-properties-common sudo vim x11-xserver-utils \
xfce4 xfce4-goodies xorgxrdp xrdp ffmpeg libaio-dev \
python3-pip python3-venv python3-dev \
vulkan-tools libvulkan1 mesa-vulkan-drivers libgl1-mesa-glx \
gnupg wget nano git-lfs xvfb ca-certificates
Note: Ensure NVIDIA drivers (recommended 525+) are installed.
sudo apt-get install -y libglvnd-dev
# if it doesn't exist
sudo mkdir -p /usr/share/vulkan/icd.d/ /usr/share/glvnd/egl_vendor.d/
# Configure Vulkan ICD if it doesn't exist
echo '{ "file_format_version" : "1.0.0", "ICD": { "library_path": "libGLX_nvidia.so.0", "api_version" : "1.2.155" } }' | sudo tee /usr/share/vulkan/icd.d/nvidia_icd.json
# Configure EGL Vendor if it doesn't exist
echo '{ "file_format_version" : "1.0.0", "ICD" : { "library_path" : "libEGL_nvidia.so.0" } }' | sudo tee /usr/share/glvnd/egl_vendor.d/10_nvidia.json
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
bash ~/miniconda.sh -b -p $HOME/miniconda
source "$HOME/miniconda/bin/activate"
conda init bash
git clone https://github.com/pablovalle/MT_of_VLAs.git
cd MT_of_VLAs
The following steps apply to both Docker and source installations.
Once everything is set up and you can access to the repository either on your local machine or inside the docker, for each VLA one conda environment will be generated and the corresponding models will be downloaded. To do so, inside environment_mount you can find one .sh file for each model. To setup the environment and download the models:
cd {this_repo/environment_mount}
./{model_name}.sh #Options: EO1, GR00T, PI0, SPATIALVLA, and OPENVLA
Note: Setting up the environment may take approximately 10–15 minutes, depending on your internet connection (Overall the fastest model in installation and execution is EO1). The model name corresponding to each conda environment is the same name but in lowercase:
eo1,gr00t,pi0and for spatialVLA and openVLA the number of parameters should be added:spatialvla-4bandopenvla-7b.
Once it finishes, you will find the model weights inside checkpoints folder and you will have the corresponding conda environment wiht the same name as the .sh file you launched. For example if you launched EO1.sh you will have a conda env called EO1.
To generate the follow-up test cases just a .sh file should be ran:
cd {this_repo/experiments}
./{follow_up_generator.sh}.sh -e <env> -m <model> [options]
Note: Pre-generated follow-up cases used in our evaluation are available in data/FollowUp. Remove, rename or move this folder if you want to regenerate test cases.
| Flag | Status | Description |
|---|---|---|
| -e | REQUIRED | Conda Environment: The name of the specific Conda environment you wish to activate for the run. |
| -m | REQUIRED | Model Name: The identifier for the model being tested (e.g., eo1, gr00t). |
| -r | OPTIONAL | Metamorphic Relations: Specifies which relations to apply (MR1 through MR5). Multiple values should be comma-separated. |
| -t | OPTIONAL | Task ID Filter: Filters the execution to specific tasks. Accepts arrays like [1,2,3] or ranges like [1-50]. |
| -d | OPTIONAL | Dataset JSON: Allows you to target specific dataset files (e.g., t-grasp_n-1000_o-m3_s-2498586606.json). |
| -h | OPTIONAL | Help: Displays the manual and all available options, then exits the script. |
Note: If no task ID is specified, to follow our experimental setup it will take all the passing test from the source test cases, so if that the case, please first run the source test cases and provide the human verification. In our case we took this data from Evaluating Uncertainty and Quality of Visual Language Action-enabled Robots.
Usage example:
./follow_up_generator.sh -e EO1 -m eo1 -r MR1,MR3 -t [1-10,15,18] -d t-grasp_n-1000_o-m3_s-2498586606.json
To execute the source test cases just a .sh file should be ran:
cd {this_repo/experiments}
./{run_source_test_cases.sh}.sh -e <env> -m <model> [options]
Note: Precomputed results for the source test cases are available in results. By default, the script will not re-run these test cases. To execute them again, you can rename, move, or remove the
/resultsfolder. Do not remove the/results/human_evalfolder
| Flag | Status | Description |
|---|---|---|
| -e | REQUIRED | Conda environment name to activate. |
| -m | REQUIRED | Specific model name to run (e.g., eo1, gr00t3). |
| -d | OPTIONAL | Dataset JSON filename(s). Accepts one or multiple values separated by commas (e.g., t-grasp_n-1000_o-m3_s-2498586606.json,t-move_n-1000_o-m3_s-2263834374.json). Default: Runs the four standard datasets (t-grasp_n-1000_o-m3_s-2498586606.json, t-move_n-1000_o-m3_s-2263834374.json, t-put-in_n-1000_o-m3_s-2905191776.json, t-put-on_n-1000_o-m3_s-2593734741.json). |
| -h | OPTIONAL | Displays the help message and exits. |
Usage example:
./run_source_test_cases.sh -e EO1 -m eo1 -d t-put-on_n-1000_o-m3_s-2593734741.json
To execute the follow-up test cases just a .sh file should be ran:
cd {this_repo/experiments}
./{run_follow_up_test_cases.sh}.sh -e <env> -m <model> [options]
Note: Precomputed results for the follow-up test cases are available in FollowUp_Results.By default, the script will not re-run these follow-up test cases. To execute them again, you can rename, move, or remove the
/FollowUp_Resultsfolder.
| Flag | Status | Description |
|---|---|---|
| -e | REQUIRED | Conda environment name to activate. |
| -m | REQUIRED | Specific model name to run (e.g., eo1, gr00t). |
| -r | OPTIONAL | Metamorphic Relations (MRs). Accepts single or multiple values separated by commas or spaces (e.g., "MR1,MR2,MR5"). Default: MR1, MR2, MR3, MR4, MR5. |
| -d | OPTIONAL | Dataset JSON filename(s). Accepts one or multiple values separated by commas (e.g., grasp,move). Default: Runs the four standard datasets (grasp, move, put-in, put-on). |
| -h | OPTIONAL | Displays the help message and exits. |
Usage example:
./run_source_test_cases.sh -e EO1 -m eo1 -r MR1,MR3 -d grasp,move
To reproduce the results reported in our paper, we provide precomputed outcomes for both the source test cases (results) and the follow-up test cases(FollowUp_Results), along with an automated analysis pipeline located in the result_analysis folder. The steps below describe how to analyze the provided results.
Note: The analysis scripts are preconfigured to generate results exactly as presented in the paper. If additional metamorphic relations (MRs) are added, the scripts must be updated to handle them correctly. Similarly, if any VLA models are added or removed, figure sizes may need adjustment.
Before running the analysis, ensure that all required results are present in their corresponding folders (this is in case you renamed, moved or removed any folder). Once ready, execute the following commands:
cd {this_repo/result_analysis}
./{result_analysis.sh}.sh -e <env>
Note: Specifying the environment is mandatory, as all required analysis libraries are included within each environment. You can perform the analysis using any of the provided environments.
Running the analysis will process the results for RQ1 and RQ2, generating a set of summary files named RQ1_results_{model_name}.xlsx, one per VLA model. Additionally, the 3rd and 4th figures from the paper, along with other relevant visualizations, will be produced in figures. The analysis will also generate the folders output_mr and output_oracle, containing the selected videos used to build the taxonomy described in RQ3.
The taxonomy itself was generated using a questionnaire, which is available here: questionnaire
RuntimeError: vk::Instance::enumeratePhysicalDevices: ErrorInitializationFailed
Some required Vulkan extension is not present. You may not use the renderer to render, however, CPU resources will be still available.
Segmentation fault (core dumped)
Follow this link to troubleshoot the issue. (Even though the doc points to SAPIEN 3 and ManiSkill3, the troubleshooting section still applies to the current environments that use SAPIEN 2.2 and ManiSkill2).
TypeError: 'NoneType' object is not subscriptable
Please also refer to the original repo or vulkan_setup if you encounter any problems.
tensorflow-2.15.0 conflicts with tensorflow-2.15.1?
The dlimp library has not been maintained for a long time, so the TensorFlow version might be out of date. A reliable solution is to comment out tensorflow==2.15.0 in the requirements file, install all other dependencies, and then install tensorflow==2.15.0 finally. Currently, using tensorflow==2.15.0 has not caused any problems.
If you find this project useful in your research, please consider cite:
@inproceedings{valle2026mt_of_vlas_icst,
title = {Metamorphic Testing of Vision-Language Action-Enabled Robots},
author = {Pablo Valle and Sergio Segura and Shaukat Ali and Aitor Arrieta},
booktitle = {Proceedings of the 19th IEEE International Conference on Software Testing, Verification and Validation (ICST 2026)},
year = {2026},
month = may,
address = {Daejeon, Republic of Korea},
organization = {IEEE},
note = {\url{https://arxiv.org/abs/2602.22579}},
url = {https://arxiv.org/abs/2602.22579}
}
This project is built with reference to the code of the following projects: Isaac-Groot, Lerobot, EO-1, SimplerEnv-OpenVLA, VLATest, and Evaluating Uncertainty and Quality of Visual Language Action-enabled Robots.
Pablo Valle and Aitor Arrieta are part of the Software and Systems Engineering research group of Mondragon Unibertsitatea (IT1519-22), supported by the Department of Education, Universities and Research of the Basque Country. Pablo Valle is supported by the Pre-doctoral Program for the Formation of Non-Doctoral Research Staff of the Education Department of the Basque Government (Grant n. PRE_2025_2_0252). Shaukat Ali is supported by the Co-tester project (No. 314544) funded by the Research Council of Norway. This work has been partially funded by the Spanish Ministry of Science, Innovation and Universities (project PID2023-152979OA-I00), funded by MCIU/AEI/10.13039/501100011033/FEDER, UE. This paper is also part of the project PID2024-156482NB-I00, funded by MICIU/AEI/10.13039/501100011033 and by the ESF+.