Ego-1K — A Large-Scale Multiview Video Dataset for Egocentric Vision

Jae Yong Lee, Daniel Scharstein, Akash Bapat, Hao Hu, Andrew Fu, Haoru Zhao, Paul Sammut, Xiang Li, Stephen Jeapes, Anik Gupta, Lior David, Saketh Madhuvarasu, Jay Girish Joshi, and Jason Wither

CVPR 2026     arXiv:2603.13741

We present Ego-1K, a large-scale collection of time-synchronized egocentric multiview videos designed to advance neural 3D video synthesis and dynamic scene understanding. The dataset contains 956 short (6.7-9.7s) egocentric videos taken with a custom rig with 12 synchronous cameras surrounding a VR headset worn by the user, for a total of 491K frames and 5.9M images. Scene content focuses on hand motions and hand-object interactions in different settings. Our dataset enables new ways to benchmark egocentric scene reconstruction methods, and presents unique challenges for existing 3D and 4D novel view synthesis methods due to high disparities and image motion caused by close dynamic objects and rig egomotion.

License

FAIR Noncommercial Research License

Getting Started

See quickstart.ipynb for a runnable walkthrough that loads metadata, streams images, and visualizes a 12-camera multiview frame.

Key Statistics

Dataset Structure

ego-1k/
├── data/
│   ├── train-<scene_id>.parquet          # Per-scene metadata index
│   └── test-<scene_id>.parquet
└── shards/
    ├── train/
    │   └── <scene_id>/
    │       ├── <scene_id>-0000.tar     # WebDataset tar shards (~1.5 GB each)
    │       ├── <scene_id>-0001.tar
    │       └── ...
    └── test/
        └── <scene_id>/
            └── <scene_id>-0000.tar

Tar Shard Contents

Each tar sample represents one frame across all 12 cameras:

<scene_id>/<frame_id:06d>.200-1.png        # Raw PNG bytes (1280x1280)
<scene_id>/<frame_id:06d>.200-2.png
...
<scene_id>/<frame_id:06d>.200-12.png
<scene_id>/<frame_id:06d>.metadata.json    # Pose, rig calibration, scene info

The metadata.json per sample contains:

Parquet Schema

Each row represents a single frame (one timestamp across all 12 cameras):

Calibration Details

The rig_calibration column contains a JSON object keyed by camera name (200-1 through 200-12), each with:

• K: 3x3 intrinsic matrix (rectified pinhole projection, 120 deg horizontal FOV)
• E: 4x4 extrinsic matrix (camera-to-device transform)

The pose column contains the 4x4 device-to-world transform, which changes per frame as the headset moves.

Usage

load_dataset returns frame-level metadata only (poses, calibration, scene info). Images are stored in WebDataset tar shards — use the webdataset library to stream them. See quickstart.ipynb for a full working example.

WebDataset (Recommended for Training)

Stream tar shards for high-throughput sequential access — no per-file API calls. See the notebook for the full decode_sample implementation. To wrap it in a PyTorch DataLoader:

dataset = wds.WebDataset(shard_urls, nodesplitter=wds.split_by_node, shardshuffle=True).map(decode_sample)
loader = torch.utils.data.DataLoader(dataset, batch_size=4, num_workers=4)

for batch in loader:
    images = batch["images"]  # (B, N_cams, 3, 1280, 1280)
    break

Parquet Metadata (Random Access)

The Parquet files contain frame-level metadata only (poses, calibration, scene info) — images are stored in the tar shards. Use the shard_name column to locate which tar file contains a given frame's images.

shard_url = f"https://huggingface.co/datasets/facebook/ego-1k/resolve/main/{example['shard_name']}"

Citation

@inproceedings{ego1k2026,
  title={{Ego-1K}: A Large-Scale Multiview Video Dataset for Egocentric Vision},
  author={Jae Yong Lee and Daniel Scharstein and Akash Bapat and Hao Hu and Andrew Fu and Haoru Zhao and Paul Sammut and Xiang Li and Stephen Jeapes and Anik Gupta and Lior David and Saketh Madhuvarasu and Jay Girish Joshi and Jason Wither},
  booktitle={CVPR},
  year={2026}
}