A Category-Aware Robustness Correction Framework for Vision-Language-Action Systems

Author: Ruohang Xu
Artefact type: MSc project source-code attachment

This repository contains the project-owned implementation of a category-aware robustness correction framework built around BitVLA and evaluated with LIBERO/LIBERO-Plus. The framework separates seven perturbation categories: camera, robot, noise, layout, language, background, and light. It includes deterministic category sampling, residual action correction, action-scale modelling, SRPO/RIPT-style post-training utilities, Screen-700 evaluation tooling, and the unified seven-category training infrastructure.

The repository is intended for source review and reproducibility. Large model weights, simulator assets, datasets, raw feature tables, videos, caches, logs, and generated experiment outputs are deliberately excluded from the Git repository.

Repository Structure

app/                         base BitVLA runtime check and local inference API
config/                      dependency specifications, setup scripts, and local patches
configs/                     recorded experiment configurations
scripts/                     training, evaluation, calibration, and diagnostic entry points
src/ript_bitvla/             residual policy and PPO components
src/srpo_bitvla/             SRPO reward and training components
src/libero_plus_unified_7/   seven-category training infrastructure
tests/                       portable unit and integration tests

Generated experiment outputs should be written outside the Git working tree. The repository intentionally does not contain a results/ or result/ directory.

Quick Verification

From the repository root:

python3 -m venv .venv-test
.venv-test/bin/pip install --upgrade pip pytest numpy PyYAML torch
.venv-test/bin/python -m pytest

Tests that require a GPU, simulator, checkpoint, dataset, or excluded experiment output are skipped when those resources are unavailable.

Recorded Environments

Two independent environments were used and should not be mixed.

CUDA experiments used the CUDA 12.8 runtime supplied by PyTorch. MuJoCo rendering used EGL and did not require a display server.

Base BitVLA Runtime Check (ROCm)

The app/ utility validates the base BitVLA runtime and provides a simple image-to-action API. It does not load the post-trained category-aware correction modules and should not be interpreted as the final corrected policy.

The ROCm setup script downloads BitVLA and the base checkpoint, applies the retained runtime patch, and installs the required environment:

bash config/setup/setup_rocm.sh
./.venv/bin/python app/main.py serve --host 127.0.0.1 --port 8000

Example request:

curl -X POST http://127.0.0.1:8000/predict \
  -F 'image=@/path/to/camera.jpg' \
  -F 'instruction=pick up the red block' \
  -F 'dataset=bridge_orig'

This runtime check uses the lxsy/bitvla-bf16 base checkpoint.

CUDA Training and Evaluation Setup

Create an isolated Python 3.12 environment on a CUDA-capable host. Install the PyTorch build first, then install simulator and SRPO dependencies without allowing them to replace PyTorch:

python3 -m venv .venv-cuda
.venv-cuda/bin/pip install --upgrade pip
.venv-cuda/bin/pip install \
  torch==2.11.0+cu128 torchvision==0.26.0+cu128 \
  torchaudio==2.11.0+cu128 triton==3.6.0 \
  --index-url https://download.pytorch.org/whl/cu128
.venv-cuda/bin/pip install -r config/requirements/runtime.txt
.venv-cuda/bin/pip install -r config/requirements/libero-smoke.txt
.venv-cuda/bin/pip install -r config/requirements/srpo.txt

Before launching simulator processes:

source config/srpo.env.example
export CUDA_PYTHON="$PWD/.venv-cuda/bin/python"
$CUDA_PYTHON -c 'import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())'
$CUDA_PYTHON -c 'import mujoco; print(mujoco.__version__)'

Files under configs/ are records of the executed server layout and may contain /root/rivermind-data/... paths. On another machine, copy the relevant YAML file, replace machine-specific project/model/dataset/manifest/checkpoint/cache/output paths, and keep the recorded seeds, category ratios, and selection rules unchanged.

Useful entry points include:

$CUDA_PYTHON scripts/libero_plus_unified_7/build_inventory.py --help
$CUDA_PYTHON scripts/libero_plus_unified_7/audit_expert_contract.py --help
$CUDA_PYTHON scripts/libero_plus_unified_7/run_stage1.py --help
$CUDA_PYTHON scripts/libero_plus_unified_7/run_dev_evaluator.py --help
$CUDA_PYTHON scripts/run_exploratory_srpo_8.py --help
$CUDA_PYTHON scripts/libero_plus/run_bitvla_screen700.py --help

Script Layout

The top-level SRPO phase scripts are retained because the executed configurations and provenance refer to those exact paths.

Selected Project Checkpoint

The selected post-training checkpoint is distributed separately from GitHub because it is a large binary artefact.

The checkpoint contains the project-owned post-training policy modules and is not a standalone Vision-Language-Action model. Evaluation also requires the matching BitVLA backbone, project source, LIBERO environment, and relevant task manifests.

The final model is referred to in the dissertation as BitVLA (post-trained). Its Screen-700 evaluation entry point is:

$CUDA_PYTHON scripts/libero_plus/run_bitvla_posttrained_screen700.py --help

The evaluation script accepts explicit paths for the base checkpoint, selected policy checkpoint, LIBERO-Plus source, manifest, and output directory. Generated outputs should remain outside the submitted Git tree.

Third-Party Sources and Large Dependencies

The project-owned implementation is under app/, src/, scripts/, and tests/. External projects and model files are dependencies and must not be presented as original project code.

Third-party repositories are not vendored in this MSc artefact. Retrieve dependencies from their upstream repositories and use the version required by the relevant setup script or experiment configuration. Local BitVLA patches are retained in config/patches/.

Submission Boundary

Included in this repository:

• project-owned source code, scripts, tests, and experiment configurations;
• recorded dependency specifications and retained local patches;
• setup and environment files needed to inspect or reproduce the implementation.

Excluded from GitHub:

• generated experiment outputs and figures;
• model weights and trained checkpoints;
• LIBERO and LIBERO-Plus simulator assets;
• RLDS datasets and raw feature arrays;
• videos, caches, logs, virtual environments, credentials, and machine-specific archives.

The selected project checkpoint is distributed separately and linked above for assessor access.

Licence and Citation

Citation metadata is provided in CITATION.cff. Project-owned material is covered by LICENSE; third-party software and model assets remain subject to their respective upstream licences.