EgoExOR: An Egocentric–Exocentric Operating Room Dataset for Comprehensive Understanding of Surgical Activities

Official code of the paper "EgoExOR: An Egocentric–Exocentric Operating Room Dataset for Comprehensive Understanding of Surgical Activities" submitted at NeurIPS 2025 Datasets & Benchmarks Track.

• Authors: Ege Özsoy, Arda Mamur, Felix Tristram, Chantal Pellegrini, Magdalena Wysocki, Benjamin Busam, Nassir Navab

Operating rooms (ORs) demand precise coordination among surgeons, nurses, and equipment in a fast-paced, occlusion-heavy environment, necessitating advanced perception models to enhance safety and efficiency. Existing datasets either provide partial egocentric views or sparse exocentric multi-view context, but do not explore the comprehensive combination of both. We introduce EgoExOR, the first OR dataset and accompanying benchmark to fuse first-person and third-person perspectives. Spanning 94 minutes (84,553 frames at 15 FPS) of two emulated spine procedures, Ultrasound-Guided Needle Insertion and Minimally Invasive Spine Surgery, EgoExOR integrates egocentric data (RGB, gaze, hand tracking, audio) from wearable glasses, exocentric RGB and depth from RGB-D cameras, and ultrasound imagery. Its detailed scene graph annotations, covering 36 entities and 22 relations (568,235 triplets), enable robust modeling of clinical interactions, supporting tasks like action recognition and human-centric perception. We evaluate the surgical scene graph generation performance of two adapted state-of-the-art models and offer a new baseline that explicitly leverages EgoExOR’s multimodal and multi-perspective signals. This new dataset and benchmark set a new foundation for OR perception, offering a rich, multimodal resource for next-generation clinical perception.

Figure: Overview of one timepoint from the EgoExoR dataset, showcasing synchronized multi-view egocentric RGB and exocentric RGB-D video streams, live ultrasound monitor feed, audio, a fused 3D point-cloud reconstruction, and gaze, hand‐pose and scene graph annotations.

🌟 Key Features

• Multiple Modalities: Each take includes RGB video, audio, eye gaze tracking, hand tracking, 3D point cloud data, and annotations, all captured simultaneously.

• Time-Synchronized Streams: All modalities are aligned on a common timeline, enabling precise cross-modal correlation (e.g. each video frame has corresponding gaze coordinates, hand positions, etc.).

• Research Applicability: EgoExOR aims to fill the gap in both egocentric and exocentric surgrical datasets, supporting development of AI assistants, skill assessment tools, and multimodal models in medical and augmented reality domains.

📂 Dataset Structure

The dataset is available in two formats:

• Individual Files: Hosted on the Hugging Face (ardamamur/EgoExOR)repository for efficient storage and access.
• Merged HDF5 File: Consolidates all data, including splits, into a single file. Should be done locally.

Individual Files (Hugging Face Repository)

Individual files are organized hierarchically by surgery type, procedure, and take, with components like RGB frames, eye gaze, and annotations stored separately for efficiency. The splits.h5 file defines the train, validation, and test splits.

• metadata/
  • vocabulary/
    • entity (Dataset: name, id)
      • Lists entities (e.g., objects, people) with their names and unique IDs.
    • relation (Dataset: name, id)
      • Lists relationships (e.g., "holding") with their names and unique IDs.
  • sources/
    • sources (Dataset: name, id)
      • Lists data sources (e.g., cameras like 'assistant', 'ultrasound', 'external') with their names and unique IDs.
      • Note: Camera IDs in eye_gaze/coordinates are mapped to this sources dataset for accurate source names. Do not use takes/<take_id>/sources/ for mapping camera IDs to get the source names, though the source names are listed in the same order.
  • dataset/
    • Attributes: version, creation_date, title
      • Provides dataset-level information, such as version number, creation date, and title.
• data/
  • <surgery_type>/
    • Directory named after the type of surgery (e.g., "MISS").
    • <procedure_id>/
      • Directory for a specific procedure.
      • takes/
        • <take_id>/
          • Directory for a specific recording (subclip) of a procedure.
          • sources/
            • Attributes: source_count (int), source_0 (e.g., 'head_surgeon'), source_1, ...
              • Metadata for take cameras/sources, mapping array indices to camera/source IDs.
              • Note: Source names are in the same order as in metadata/sources, but for camera/source ID mapping (in gaze), use metadata/sources to get accurate source names.
          • frames/
            • rgb (Dataset: [num_frames, num_cameras, height, width, 3], uint8)
              • Synchronized video frames with dimensions: number of frames, number of cameras, height, width, and 3 color channels.
          • eye_gaze/
            • coordinates (Dataset: [num_frames, num_ego_cameras, 3], float32)
              • Eye gaze data from Egocentric devices with dimensions: number of frames, number of ego cameras, and 3 values (camera/source ID, x-coordinate, y-coordinate).
              • Invalid gaze points are marked as [-1., -1.].
              • Note: The camera_id in the last dimension must be mapped to metadata/sources for the correct source name, not to takes/<take_id>/sources/.
          • eye_gaze_depth/
            • values (Dataset: [num_frames, num_ego_cameras], float32)
              • Depth values for eye gaze in meters, synchronized with eye_gaze/coordinates (can use camera/source ID from coordinates).
              • Defaults to 1.0 if depth data is unavailable.
          • hand_tracking/
            • positions (Dataset: [num_frames, num_ego_cameras, 17], float32)
              • Hand tracking data from egocentric devices with dimensions: number of frames, number of ego cameras, and 17 values (camera ID + 8 keypoints for left hand + 8 keypoints for right hand, including wrist, palm, and normals).
              • Invalid points are marked with NaN.
          • audio/ (Optional)
            • waveform (Dataset: [num_samples, 2], float32)
              • Full stereo audio waveform with dimensions: number of samples and 2 channels (left, right).
            • snippets (Dataset: [num_frames, samples_per_snippet, 2], float32)
              • 1-second stereo audio snippets aligned with frames, with dimensions: number of frames, samples per snippet, and 2 channels.
          • point_cloud/
            • coordinates (Dataset: [num_frames, num_points, 3], float32)
              • Merged 3D point cloud coordinates from external cameras, with dimensions: number of frames, number of points, and 3 coordinates (x, y, z).
            • colors (Dataset: [num_frames, num_points, 3], float32)
              • RGB colors for point cloud points (0-1 range), with dimensions: number of frames, number of points, and 3 color channels.
          • annotations/
            • frame_idx
              • rel_annotations (Dataset: [n_annotations_per_frame, 3], object (byte string))
                • Text-based scene graph annotations (e.g., "head_surgeon holding scalpel") for each frame.
              • scene_graph (Dataset: [n_annotations_per_frame, 3], float32)
                • Tokenized annotations using integer mappings from metadata/vocabulary, representing relationships in a structured format.
• splits.h5
  • A standalone file defining the dataset splits (train, validation, test).
  • Contains columns: surgery_type, procedure_id, take_id, frame_id
    • surgery_type: Type of surgical procedure (e.g., "appendectomy").
    • procedure_id: Unique identifier for a specific procedure.
    • take_id: Identifier for a specific recording (subclip) of a procedure.
    • frame_id: Identifier for individual frames within a take.

Merged Dataset File (Locally)

The merged dataset file consolidates all data from the individual files into a single file, including the splits defined in splits.h5. This file follows the same structure as above, with an additional splits/ directory that organizes the data into train, validation, and test subsets.

• splits/
  • train, validation, test
    • Each split is a dataset with columns: surgery_type, procedure_id, take_id, frame_id
      • Links to the corresponding data in the data/ directory for easy access during machine learning tasks.

⚙️ Efficiency and Usability

• Efficiency:
  • HDF5 Format: Ideal for large, complex datasets with hierarchical organization and partial loading.
  • Compression: gzip reduces file size, critical for video and point cloud data.
  • Chunking: Enables efficient access to specific frame ranges, supporting sequence-based model training.
• Usability:
  • Logical Structure: Hierarchical organization (data/surgery/procedure/take/modality) simplifies navigation.
  • Embedded Metadata: Source mappings and vocabularies enhance self-containment.
• Scalability: Easily accommodates new surgeries or clips by adding groups to the existing hierarchy.

📜 License

Released under the Apache 2.0 License, permitting free academic and commercial use with attribution.

---

📚 Citation

@inproceedings{NEURIPS2025_5e3ffa2c,
  author = {\"{O}zsoy, Ege and Mamur, Arda and Tristram, Felix and Pellegrini, Chantal and Wysocki, Magdalena and Busam, Benjamin and Navab, Nassir},
  booktitle = {Advances in Neural Information Processing Systems},
  editor = {D. Belgrave and C. Zhang and H. Lin and R. Pascanu and P. Koniusz and M. Ghassemi and N. Chen},
  pages = {},
  publisher = {Curran Associates, Inc.},
  title = {EgoExOR: An Ego-Exo-Centric  Operating Room Dataset for Surgical Activity Understanding},
  url = {https://proceedings.neurips.cc/paper_files/paper/2025/file/5e3ffa2c53dce23986ca0f8d1d2bbc7e-Paper-Datasets_and_Benchmarks_Track.pdf},
  volume = {38},
  year = {2025}
}

---

## 🤝 Contributing

Contributions are welcome! Submit pull requests to improve loaders, add visualizers, or share benchmark results.

---


*Dataset URL: [ardamamur/EgoExOR](https://huggingface.co/datasets/ardamamur/EgoExOR)*  
*Last Updated: May 2025*