SparseVideoNav Datasets

This repository contains the real-world navigation datasets released with OpenDriveLab/SparseVideoNav:

• BVN: Beyond-the-View Navigation.
• IFN: Instruction-Following Navigation.

Project links:

• Project page: https://opendrivelab.com/SparseVideoNav
• GitHub: https://github.com/OpenDriveLab/SparseVideoNav
• Paper: https://arxiv.org/abs/2602.05827

Dataset Summary

SparseVideoNav studies real-world vision-language navigation with sparse future video generation. The datasets contain language instructions, RGB frame sequences, and low-level navigation actions. The number of actions matches the number of RGB frames for every released episode.

This repository version contains the processed IFN and BVN subsets used by SparseVideoNav. The complete dataset contains about 140 hours; due to regional policy restrictions, the currently open-sourced portion is approximately 121.74 hours.

Duration is computed as num_frames / 4 / 3600.

Repository Structure

Images are stored in compressed tar shards to avoid hundreds of thousands of small files in the Hugging Face repository. Each shard preserves the original relative paths.

.
├── README.md
├── assets/
│   └── dataset_mosaic.png
├── frames/
│   ├── bvn/
│   │   ├── annotations.json
│   │   ├── data.jsonl
│   │   ├── merge_info.json
│   │   ├── shard_manifest.jsonl
│   │   └── shards/
│   │       ├── bvn-00000.tar.zst
│   │       └── ...
│   └── ifn/
│       ├── annotations.json
│       ├── data.jsonl
│       ├── merge_info.json
│       ├── shard_manifest.jsonl
│       └── shards/
│           ├── ifn-00000.tar.zst
│           └── ...
└── raw_videos/
    ├── README.md
    ├── manifest.json
    ├── bvn/
    │   ├── metadata.jsonl
    │   └── <episode_id>.mp4
    └── ifn/
        ├── metadata.jsonl
        └── <episode_id>.mp4

Current shard counts:

Privacy-Processed Episode Videos

The episode video release is temporarily unavailable while video-only MP4 files are rebuilt and reuploaded. The frames/ release remains available.

Data Format

Each line in frames/bvn/data.jsonl or frames/ifn/data.jsonl is an episode-level JSON object.

Each action object contains:

Example:

{
  "dataset": "ifn",
  "episode_id": "<episode_id>",
  "instruction": "please go along with the rail until you are near by a red cone.",
  "num_frames": 177,
  "num_actions": 177,
  "rgb_dir": "images/<episode_dir>/rgb",
  "actions": [{"dx": 0.0429, "dy": -0.0311, "dyaw": 0.0271}]
}

annotations.json stores the annotation records with the core fields id, video, actions, and instructions. shard_manifest.jsonl stores shard-level metadata, including the shard path, episode ids, raw byte size, compressed byte size, and frame count.

Usage

Load episode metadata with Hugging Face Datasets:

from datasets import load_dataset

bvn = load_dataset("OpenDriveLab/SparseVideoNav", "bvn")
ifn = load_dataset("OpenDriveLab/SparseVideoNav", "ifn")

Download and inspect shards:

tar -I zstd -tf frames/ifn/shards/ifn-00000.tar.zst | head
tar -I zstd -xf frames/ifn/shards/ifn-00000.tar.zst

After extraction, image paths resolve to paths such as:

images/<episode_dir>/rgb/000.jpg

License

The dataset is released under CC BY-NC-SA 4.0.

Citation

@article{zhang2026sparse,
  title={Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation},
  author={Zhang, Hai and Liang, Siqi and Chen, Li and Li, Yuxian and Xu, Yukuan and Zhong, Yichao and Zhang, Fu and Li, Hongyang},
  journal={arXiv preprint arXiv:2602.05827},
  year={2026}
}