Compact Representations for World Models

Project in Computational Science

References

• Article: Planning with Reasoning using Vision Language World Model
• Article: Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere

Running the Project on BerzeLiUs

Configure SSH key with GitHub

Skip, if already completed.

1. To create a key, run ssh-keygen -t ed25519. Select the default name and choose a password (technically optional).
2. Run cat ~/.ssh/id_ed25519.pub to reveal your public key. Upload it to github here.

Download the project

/proj/sciml/users/your_username_here/ # Open your user directory in the project
git clone git@github.com:AlbinLiljefors/cr4wm.git # Clone the repo

Enter your HuggingFace token

Within the repo top directory (/proj/sciml/users/your_username_here/cr4wm/), create a file called .env and write

export HF_TOKEN=hf_XXXXXXXXXXXXX

Build the Apptainer Enviroment

Run apptainer build ./berzelius.sif ./berzelius.def to build the container. This may take 5-10 minutes.

Scheduling a job

1. Open the prod directory cd prod.
2. Schedule the job by running sbatch cr4wm.sh. This will create an array of jobs.
3. To inspect their status, run watch squeue -u <your_username>. This will update every 2 seconds.

Note: The dataset is predownloaded onto the shared directory /proj/sciml/share/cr4wm/data/. Nothing needs to be done with this.

Changing the config

By default, this will create 20 jobs, running 10 videos ecah. To change the number of jobs, edit this line #SBATCH --array 0-19. To edit the number of videos per job, change STEP=XX.

Creating an venv for postprocessing

python -m venv postprocessing
source postprocessing/bin/activate
pip install pandas requests

Postprocessing the results

Run python postprocessing.py to calculate the accuracy. Remeber to activate the enviroment with source postprocessing/bin/activate if not already done.

When done, rename the results folder to something else, i.g. results_n with incrementing n, so the results does not get overwritten by other runs.