embodied-eval

Why embodied-eval

🚀 TODO

• Open-EQA
• • EM-EQA
• • A-EQA
• GOAT-Bench

🔨 Setup

1. Clone this repository & Install packages

git clone https://github.com/LongXinKou/embodied-eval.git
cd embodied-eval
conda create -n embodied-eval python==3.10
conda activate embodied-eval
pip install -r requirements.txt

Some model backends require additional model-specific packages that are not installed by the base requirements file. See embodied_eval/models/README.md before running adapters such as roboannotatorx, vila, robopoint, or embodiedgpt.

🎁 Benchmark

Custom MP4 video inference

Use --custom_video_path to run result generation on a local .mp4 file that is not part of a benchmark dataset. This mode skips benchmark loading and metric evaluation because custom videos do not include ground-truth answers.

For --model roboannotatorx, install the extra RoboAnnotatorX backend first:

pip uninstall -y llava
pip install --no-deps "llava @ git+https://github.com/haotian-liu/LLaVA.git@1619889c712e347be1cb4f78ec66e7cf414ac1a6"
git clone https://github.com/LongXinKou/RoboannotatorX.git
cd RoboannotatorX
pip install -e . --no-deps

python -m embodied_eval \
    --model roboannotatorx \
    --model_args model_name_or_path=/path/to/roboannotatorx,model_base=/path/to/vicuna-13b-v1.5,image_processor=openai/clip-vit-large-patch14-336 \
    --custom_video_path /path/to/demo.mp4 \
    --custom_prompt "What actions does the robot perform in temporal order?" \
    --custom_generation_kwargs max_new_tokens=512,temperature=0 \
    --output_path ./logs/custom_video

If --custom_prompt is omitted, the runner uses a default robot-demonstration annotation prompt. For multiple prompts, pass a .txt, .json, or .jsonl file:

python -m embodied_eval \
    --model roboannotatorx \
    --model_args model_name_or_path=/path/to/roboannotatorx,model_base=/path/to/vicuna-13b-v1.5,image_processor=openai/clip-vit-large-patch14-336 \
    --custom_video_path /path/to/demo.mp4 \
    --custom_prompts_file ./prompts.txt \
    --output_path ./logs/custom_video