SAGE-3D VLN Data: Vision-Language Navigation Dataset with Hierarchical Instructions

Paper | Project Page | Code

A comprehensive VLN dataset featuring 2 million trajectory-instruction pairs across 1,000 indoor scenes, with hierarchical instruction design covering high-level semantic goals to low-level control commands.

---

📢 News

• 2025-12-15: Released SAGE-3D VLN Data with 2M trajectory-instruction pairs and hierarchical instruction design.

---

📋 Overview

Vision-and-Language Navigation (VLN) requires agents to follow natural language instructions to navigate in complex environments. However, existing VLN datasets primarily focus on simple "A-to-B" navigation without causal context or diverse instruction complexity levels.

SAGE-3D VLN Data addresses these limitations by providing:

• 2M trajectory-instruction pairs across 1,000 diverse indoor scenes
• Hierarchical instruction design from high-level semantic goals to low-level control primitives
• Causal instruction support (e.g., "I'm thirsty, get water from the fridge")
• Three generalization test splits: Scene-Unseen, Trajectory-Unseen, Instruction-Unseen
• Training data ready for action and image generation

Key Features

---

🗂️ Dataset Structure

SAGE-3D_VLN_Data/
├── test/                                    # Test set
│   ├── Trajectory_test/                     # High-level VLN test
│   │   ├── Trajectory_test_Instruction_Unseen/
│   │   │   ├── 839992/
│   │   │   │   └── test_trajectories_overall_*.json
│   │   │   └── ...
│   │   ├── Trajectory_test_Scene_Unseen/
│   │   │   └── ...
│   │   └── Trajectory_test_Trajectory_Unseen/
│   │       └── ...
│   ├── Trajectory_test_low-level/           # Low-level test
│   │   └── ...
│   └── Trajectory_test_no-goal/             # No-goal test
│       └── ...
└── train/                                   # Training set
    └── Trajectory_train/
        ├── 839873/
        │   └── train_trajectories_overall_*.json
        └── ...

---

📊 Data Description

Test Set

The test set is organized into three task types:

1. High-Level VLN (Trajectory_test/)

Standard vision-language navigation with semantic, causal, and spatial reasoning:

Instruction Types:

• Add_Object: Adds causal object manipulation context
  • Example: "Take an ornament from the display cabinet and showcase it near the decorative wall."
• Scenario_Driven: Embeds human-centered scenarios or goals
  • Example: "I'm thirsty, please bring me a drink from the fridge."
• Relative_Relationship: Uses spatial relations to identify targets
  • Example: "Move toward the wall design situated beside the showcase sections."
• Attribute-based: Describes targets using perceivable attributes
  • Example: "Find an empty table in the dining hall."
• Area-based: Directs to functional areas rather than specific objects
  • Example: "Go to the showroom's wall design area."

Test Splits:

• Instruction-Unseen: Unseen instruction phrasings for seen trajectories
• Trajectory-Unseen: Unseen trajectory patterns in seen scenes
• Scene-Unseen: Completely unseen scene types

2. Low-Level VLN (Trajectory_test_low-level/)

Focus on low-level control and kinematic evaluation:

Instruction Types:

• Base-Actions: Fundamental control behaviors (forward/backward steps, rotations)
  • Example: "Move forward two steps." / "Turn 90 degrees to the right."
• Single-Goal: Direct point-to-point navigation without semantic context
  • Example: "Go from the kitchen to the living room." / "Walk to the sofa."

3. No-goal VLN (Trajectory_test_no-goal/)

Goal-free exploration to assess environmental understanding and collision avoidance:

• No explicit navigation target
• Evaluates exploration coverage and navigation safety
• Test ends upon collision or maximum steps reached

Training Set

The training set contains 929 scene trajectory files with rich instruction diversity:

• 929 scenes with multiple trajectories per scene
• Average ~100 trajectories per scene
• Multiple instructions per trajectory (5-7 instruction types)
• Ready for action and image generation (see Further Processing)

JSON Data Format

Each trajectory file contains (data example):

{
  "Data_2set_metaData_2": {
    "name": "GVLN"
  },
  "scenes": [
    {
      "scene_id": "839992",
      "scene_name": "839992",
      "samples": [
        {
          "trajectory_id": "0",
          "instructions": [
            {
              "instruction_type": "Add_Object",
              "start": "display_cabinet_1",
              "end": "wall_design_2",
              "generated_instruction": "Take an ornament...",
              "scene_id": "0107_839992"
            }
          ],
          "points": [
            {
              "point": "0",
              "position": [x, y, z],
              "rotation": [qx, qy, qz, qw]
            }
          ]
        }
      ]
    }
  ]
}

Key Fields:

• instruction_type: Type of instruction (Add_Object, Scenario_Driven, etc.)
• start / end: Starting and ending object names
• generated_instruction: Natural language instruction
• points: Waypoints along the trajectory with 3D position and rotation

---

🔧 Usage

Running Benchmark Tests

For evaluation on test sets, refer to the SAGE-3D GitHub Repository for detailed benchmark running instructions.

Quick Example:

# Start VLM server (e.g., NaVILA)
python scripts/vlm_server_multigpu.py --model_path /path/to/model --port 54321

# Run high-level VLN test
/path/to/isaac-sim/python.sh \
    Code/benchmark/environment_evaluation/run_benchmark.py \
    --scene_usd_path /path/to/InteriorGS_usda \
    --batch_test_dir /path/to/SAGE-3D_VLN_Data/test/Trajectory_test/Trajectory_test_Instruction_Unseen \
    --map_path /path/to/semantic_maps \
    --output_root /path/to/results \
    --task-type vln \
    --vlm-port 54321

---

🏗️ Further Processing

Training Data Processing

The training trajectory data needs further processing to generate:

1. Action sequences (MOVE_FORWARD, TURN_LEFT, TURN_RIGHT, STOP)
2. RGB images rendered at waypoints using Isaac Sim

Step 1: Generate Action Sequences

python Code/data_pipeline/training_data_construction/generate_actions.py \
    --input-dir /path/to/SAGE-3D_VLN_Data/train/Trajectory_train \
    --output-dir /path/to/output/actions

Step 2: Render RGB Images

# Use Isaac Sim's Python interpreter
/path/to/isaac-sim/python.sh \
    Code/data_pipeline/training_data_construction/generate_images.py \
    --input-dir /path/to/SAGE-3D_VLN_Data/train/Trajectory_train \
    --usd-root /path/to/InteriorGS_usda \
    --output-dir /path/to/output/images \
    --action-root /path/to/actions

For complete pipeline and detailed parameters, see the SAGE-3D GitHub Repository.

---

📈 Statistics

Test Set Statistics

Note: High-Level VLN task is divided into three domain-aware test splits. Each split contains approximately equal number of scenes and trajectories.

Training Set Statistics

Instruction Type Distribution

---

🔗 Related Datasets

This dataset is part of the SAGE-3D project:

1. InteriorGS: Original 3DGS scenes with semantic annotations
   → spatialverse/InteriorGS

2. SAGE-3D InteriorGS USDZ: USDZ format for Isaac Sim
   → spatialverse/SAGE-3D_InteriorGS_usdz

3. SAGE-3D Collision Mesh: Physics-enabled collision bodies
   → spatialverse/SAGE-3D_Collision_Mesh

4. SAGE-3D VLN Data (This dataset): Navigation trajectories and instructions
   → spatialverse/SAGE-3D_VLN_Data

---

📄 License

This dataset is released under CC-BY-4.0.

---

📜 Citation

If you use SAGE-3D VLN Data in your research, please cite our paper:

@inproceedings{miao2026towards,
  title = {Towards Physically Executable {3D} Gaussian for Embodied Navigation},
  author = {Miao, Bingchen and Wei, Rong and Ge, Zhiqi and Sun, Xiaoquan and Gao, Shiqi and Zhu, Jingzhe and Wang, Renhan and Tang, Siliang and Xiao, Jun and Tang, Rui and Li, Juncheng},
  booktitle = {The Fourteenth International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=HB6KvsqcAn}
}

Please also cite the InteriorGS dataset:

@misc{InteriorGS2025,
  title        = {InteriorGS: A 3D Gaussian Splatting Dataset of Semantically Labeled Indoor Scenes},
  author       = {SpatialVerse Research Team, Manycore Tech Inc.},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/datasets/spatialverse/InteriorGS}}
}

---