WorldDreamer RoboCasa365 Evaluation

This repository contains the inference code and RoboCasa365 evaluation runner for a trained WorldDreamer policy checkpoint.

The evaluation is a closed-loop simulation rollout. A local policy server loads the WorldDreamer checkpoint on GPU, while a RoboCasa365 rollout process creates the Gym environments, renders observations, sends batched observations to the server, receives actions, and steps the simulator.

RoboCasa365 Gym envs
  -> RGB observations + robot state + language + camera geometry
  -> local WorldDreamer policy server
  -> predicted action chunks
  -> RoboCasa365 env.step(...)
  -> success metrics, CSV summaries, optional videos

Checkpoint was uploaded to the Hugging Face: world_dreamer

Repository Layout

configs/task_sets.yaml             RoboCasa365 task groups and horizons
scripts/eval_robocasa365.sh        single-shard evaluation launcher
scripts/eval_target50_3gpu.sh      multi-GPU target50 launcher
scripts/merge_robocasa365_shards.py
src/wmd_eval_runtime/              rollout, transport, and RoboCasa365 wrappers
src/world_multidreamer/            WorldDreamer model and policy code

Environments

Use Python 3.10 for the WorldDreamer model server environment.

uv sync

For CUDA environments:

uv sync --extra cuda

RoboCasa365 should be installed in a separate simulation environment with RoboCasa, robosuite, MuJoCo, ffmpeg, and EGL headless rendering support. The evaluation scripts use SERVER_PYTHON for the model server and PY365 for the RoboCasa365 rollout client.

Before running simulation:

export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl
export WMD_ROBO_RAYROPE_GEOM_STRICT=1

Checkpoint Layout

MODEL_PATH can point to either a local checkpoint directory or a Hugging Face repository id. A local checkpoint directory should contain:

config.json
model.safetensors.index.json
model-*.safetensors
processor_config.json
statistics.json
embodiment_id.json
assets/
backbone/

RoboCasa365 Setup

The task groups and per-task rollout horizons are defined in:

configs/task_sets.yaml

target50 expands to the 50-task target set:

atomic_seen + composite_seen + composite_unseen

The standard evaluation settings are:

split=target
n_envs=5
n_episodes=50
n_action_steps=8

The runner uses gym.make("robocasa/<task>", split="target", seed=...) and does not require modifying RoboCasa source code. Per-task horizons are read from configs/task_sets.yaml by default.

Camera Geometry

WorldDreamer uses Robo-RayRoPE camera geometry during evaluation. The RoboCasa365 wrapper adds these observation fields for each environment:

robo_rayrope_K
robo_rayrope_T_base_cam

The wrapper computes camera intrinsics and extrinsics from robosuite/MuJoCo at runtime. With WMD_ROBO_RAYROPE_GEOM_STRICT=1, evaluation fails immediately if the camera geometry cannot be read.

Target50 Evaluation

This launcher splits the 50 target tasks across three GPUs. Each shard starts one local policy server and one RoboCasa365 rollout process.

export MODEL_PATH=/path/to/WorldDreamer-RoboCasa365
export SERVER_PYTHON=/path/to/model-env/bin/python
export PY365=/path/to/robocasa365-env/bin/python
export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl
export WMD_ROBO_RAYROPE_GEOM_STRICT=1

bash scripts/eval_target50_3gpu.sh \
  --model-path "$MODEL_PATH" \
  --task-yaml configs/task_sets.yaml \
  --task-set target50 \
  --split target \
  --n-episodes 50 \
  --n-envs 5 \
  --n-action-steps 8 \
  --gpus "0 1 2" \
  --ports "20201 20202 20203" \
  --output-root output/robocasa365_eval

For a different machine, set --gpus and --ports to lists with the same length.

Outputs

Each run creates an experiment directory under --output-root.

summary_shard*of*.csv              per-shard task success rates
server_shard*of*.log               model server logs
videos/                            optional rollout videos

After all shards finish, merge the summaries:

python scripts/merge_robocasa365_shards.py \
  --exp-dir output/robocasa365_eval/<run_dir> \
  --task-yaml configs/task_sets.yaml

Merged outputs:

summary_merged.csv                 per-task success rates
summary_merged.json                run metadata and aggregate score
summary_by_group.csv               atomic_seen, composite_seen, composite_unseen