This guide walks through setting up the environment for training and inference with ST-LLM, including dependencies like FlashAttention, LLaVA-NeXT, MinkowskiEngine, Openscene, PointNet++, Mask2Former, Semantic-SAM, and Deformable DETR.

Step 1. Create Conda Environment

export Main=$(pwd)
conda create -n stllm python=3.9
conda activate stllm

Step 2. Install PyTorch (2.4.1) with CUDA (11.8)

MinkowskiEngine requires CUDA version < 12.0.

conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=11.8 -c pytorch -c nvidia

Step 3. Install Build Tools

pip install ninja setuptools==69.5.1 

✅ Ensure ninja is in your PATH. You can check this with:

which ninja

Step 4. Install FlashAttention (v2.5.7)

TMPDIR=/tmp \
PIP_CACHE_DIR=/tmp/pip-cache \
TORCH_EXTENSIONS_DIR=/tmp/torch-extensions \
TRITON_CACHE_DIR=/tmp/triton-cache \
MAX_JOBS=4 \
pip install -v flash-attn==2.5.7 --no-build-isolation

Step 5. Install LLaVA-NeXT

cd LLaVA-NeXT
pip install -e ".[train]"

Step 6. Install OpenScene

conda install conda-forge::openexr
conda install openblas-devel -c anaconda # Please find a way to install openblas

# Install MinkowskiEngine
git clone https://github.com/NVIDIA/MinkowskiEngine.git
cd MinkowskiEngine
python setup.py install --blas=openblas

Step 7. Install PointNet++ and accelerated giou from source:

cd LLaVA-NeXT/llava/model/openscene/third_party/pointnet2
python setup.py install
cd ../utils
python cython_compile.py build_ext --inplace

Step 8. Install Python Dependencies

cd $Main
pip install -r requirements.txt

Step 9. Install Deformable DETR

git clone https://github.com/fundamentalvision/Deformable-DETR.git
cd Deformable-DETR
cd ./models/ops
sh ./make.sh
cd $Main

Step 10. Install Semantic-SAM and Mask2Former Please install Semantic-SAM follow their instructions on their repo.

# Install Mask2Former
cd Semantic-SAM
git clone https://github.com/facebookresearch/Mask2Former.git
cd Mask2Former/mask2former/modeling/pixel_decoder/ops
sh make.sh

Step 11. Substitute modeling_utils.py in Transformers package.

# Find the path to Tranformers
tf_path=$(python -c "import transformers; import os; print(os.path.dirname(transformers.__file__))")
echo "$tf_path"
cd $Main
mv modeling_utils.py "$tf_path"

We provide a simple script to run inference on a sample REA QA example. Make sure the pretrained weights and sample data are properly downloaded.

(coming soon, currently eval code)
python LLaVA-NeXT/llava/train/inference.py

Before using our data or running any code, please download the EPIC-KITCHENS dataset (RGB video frames).
We use the downsampled version of the videos for all processing.

• Official website: https://epic-kitchens.github.io/epic-fields/
• Download the RGB frames (downsampled version) following their instructions.

📁 REA 3D Data

We release the 3D data on Google Drive. This package contains:

• Point clouds reconstructed for each scene
• Camera poses for the egocentric action video
  (32 uniformly sampled frames per clip)

Instructions to place the data: To be provided.

bash LLaVA-NeXT/scripts/video/train/stllm_rea_train.sh

cd baseline_inference

To Evaluate LLaVA-Video-7B-Qwen2

python llava_video_qwen2_inference.py --start 0 --end -1 --cuda 0 \
    --json_path ../REA_dataset/qa_val_1757_v20.json \
    --rgb_dir /path/to/EPIC-KITCHENS/rgb \
    --pretrained lmms-lab/LLaVA-Video-7B-Qwen2 \
    --model_name llava_qwen \
    --scene_level_recon True \
    --pcd_folder /path/to/epic-kitchens-vggt-anyloc-val-scene

To evaluate LLaVA-OV-Qwen2-7B

python llava_video_qwen2_inference.py --start 0 --end -1 --cuda 0 \
    --json_path ../REA_dataset/qa_val_1757_v20.json \
    --rgb_dir /path/to/EPIC-KITCHENS/rgb \
    --pretrained lmms-lab/llava-onevision-qwen2-7b-ov \
    --model_name llava_qwen \
    --scene_level_recon True \
    --pcd_folder /path/to/epic-kitchens-vggt-anyloc-val-scene

To evaluate Qwen2-VL-7B-Instruct
Modify the paths in the main function and run

python qwen2vl7binstruct_inference.py