SonoScene360: The SonoWorld Evaluation Dataset

This repository hosts SonoScene360, the real-world evaluation dataset introduced in SonoWorld: From One Image to a 3D Audio-Visual Scene (CVPR 2026). SonoWorld generates a 3D audio-visual scene with spatialized sound from a single input image.

[Paper] [Project website] [Official code]

The current dataset snapshot contains 10 scenes, 34 calibrated microphone poses, and 68 selected SN3D-normalized first-order Ambisonics (FOA) recordings.

Dataset statistics

Download

Download the complete repository with huggingface_hub:

from huggingface_hub import snapshot_download

dataset_root = snapshot_download(
    repo_id="DerongJin/SonoScene360",
    repo_type="dataset",
)

Alternatively, use the Hugging Face CLI:

hf download DerongJin/SonoScene360 \
  --repo-type dataset \
  --local-dir SonoScene360

After downloading, use data/metadata.json as the dataset-wide index. All paths stored in the index are relative to the repository root.

1. Dataset contents

Directory structure

data/
|-- metadata.json
`-- <scene_id>/
    |-- audio/
    |   `-- <mic_id>/
    |       `-- <sample_id>/
    |           `-- foa.wav
    |-- images/
    |   |-- panorama.jpg
    |   |-- panorama_outpainted.jpg
    |   `-- <mic_id>/
    |       `-- rendered.jpg
    |-- metadata/
    |   |-- known_sources.json
    |   `-- <mic_id>/
    |       `-- metadata.json
    `-- video/                  # reserved; not included in the public release yet

Each scene provides the following data:

• images/panorama_outpainted.jpg is the outpainted equirectangular panorama used as the scene input to our method. images/panorama.jpg retains the original, non-outpainted panorama for reference.
• audio/<mic_id>/<sample_id>/foa.wav stores one audio sample. A microphone may have one or more selected samples. Every released file is approximately 10 seconds of four-channel, 48 kHz, 16-bit PCM, SN3D-normalized FOA audio in W, Y, Z, X channel order.
• metadata/known_sources.json stores the important semantic sound classes shared by the entire scene.
• metadata/<mic_id>/metadata.json stores the two 2D microphone annotations, microphone rotation, and text-labelled sound-source directions for one microphone pose. These fields are explained in Microphone metadata and Microphone calibration.
• images/<mic_id>/rendered.jpg is a novel-view panorama rendered by our method at the corresponding microphone position. We use this image as the visual input for image-conditioned baselines because those baselines do not themselves render a novel view at the microphone position.

Dataset-level index

data/metadata.json is the entry point for loading the release. All stored paths are relative to the repository root.

Microphone metadata

Each data/<scene_id>/metadata/<mic_id>/metadata.json has the following form:

{
  "2d_points": {
    "mic_center_2d": {"x": 8053, "y": 2929},
    "mic_location_2d": {"x": 8070, "y": 3571}
  },
  "apriltag": {
    "rotation": [
      [-0.626414, -0.777761, -0.051903],
      [0.027733, 0.044306, -0.998633],
      [0.778997, -0.626997, -0.006184]
    ]
  },
  "mic_id": "mic_08",
  "scene_id": "fountain-multi",
  "sound_source_annotations": [
    {"direction": "right", "text_label": "fountain"}
  ]
}

• mic_location_2d is the manually annotated ground sticker directly below the microphone.
• mic_center_2d is the manually annotated microphone center. It was called the elevation point during data collection because, together with the ground point, it determines the microphone elevation angle.
• Both points are integer pixel coordinates in the original calibration panorama: x increases to the right and y increases downward. They are not normalized coordinates.
• apriltag.rotation is a 3 x 3 world-to-microphone rotation matrix. Its exact convention is described below.
• sound_source_annotations contains a semantic label and one of front, left, right, or behind for each annotated source.

Each data/<scene_id>/metadata/known_sources.json is shared by every microphone in that scene:

{
  "known_sources": ["fountain"]
}

The canonical scene-level values are:

Example visualization

The following audit sheet shows fountain-multi / mic_08 / sample_001:

The left panel uses images/mic_08/rendered.jpg as its background. Cyan shows the annotated, text-labelled source direction, while magenta shows the dominant direction estimated from the FOA signal. The right panel shows relative FOA directional energy, normalized per recording to 0 dB and clipped at -30 dB; its magenta point marks the same estimated peak.

For all equirectangular visualizations, image azimuth runs from -180 degrees on the left to +180 degrees on the right: left is -90 degrees, front is 0 degrees, and right is +90 degrees. Image elevation runs from +90 degrees at the top to -90 degrees at the bottom.

2. Release status and TODO

The calibration panoramas and videos are not part of the public release yet because some frames contain bystanders and must be privacy-filtered first. The data preparation and calibration scripts are also not released yet.

The cropped calibration examples below are included only to document the annotation and pose-estimation procedure.

3. Microphone calibration

We estimate microphone orientation and position separately. Orientation comes from AprilTag pose estimation. Position is reconstructed from two manual 2D annotations and the depth rendered for the particular scene reconstruction being evaluated. The figures below use fountain-multi / mic_08 as an example.

3.1 Manual 2D position annotations

A sticker is placed on the ground directly below the microphone. In the input-camera calibration panorama, we manually annotate:

1. the center of that ground sticker (mic_location_2d); and
2. the physical center of the microphone (mic_center_2d).

The left panel is the unannotated crop. The right panel shows the microphone center in magenta and the ground sticker in cyan. The two pixels define rays from the input camera toward two vertically aligned points.

3.2 AprilTag orientation

An AprilTag rigidly attached to the microphone rig provides its orientation. We project the equirectangular calibration image into perspective views, detect the tag in a selected view, estimate its pose there, and transform that orientation back into the panorama world frame.

The left panel shows the selected perspective view and detected tag boundary. The right panel is a closer view of the pose axes used for the rotation estimate. We retain only the resulting rotation in the released metadata; the raw AprilTag translation is discarded.

The stored apriltag.rotation is the rotation block R from est_extrinsics_new. It is a world-to-camera (w2c) rotation in the final microphone/object-camera convention, not a c2w rotation. A world-space point is transformed by

p_mic = R_w2m p_world + t_w2m.

Here, “object frame” means the microphone-local frame after converting the raw AprilTag axes. The conventions are:

Consequently, the microphone frame uses OpenCV camera axes (+X right, +Y down, +Z forward). In the panorama/world frame, -X is the input-camera forward direction, +Y is right, and +Z is up.

3.3 Reconstructing translation from depth

We do not publish a fixed 3D microphone location or translation. Instead, the evaluated reconstruction renders an equirectangular depth map at the input-camera pose, and the microphone center is reconstructed from that depth map plus the two released 2D points. This matters because different reconstruction runs can produce different depth at the same annotated pixel.

For an image of width W and height H, normalize a pixel as u = x / W and v = y / H. The calibration code converts it to polar angle phi and panorama angle theta using

phi   = pi v
theta = 2 pi (1 - u)
ray(phi, theta) = [cos(theta) sin(phi), sin(theta) sin(phi), cos(phi)].

Let (phi_g, theta_g) denote the ground sticker, (phi_c, theta_c) the microphone center, and d_g the rendered depth sampled at the ground sticker. Assuming that the microphone center is vertically above the sticker, its ray distance is

d_c = d_g sin(phi_g) / sin(phi_c),
C_world = d_c ray(phi_c, theta_c).

Once the run-specific microphone center C_world is available, the w2c translation is

t_w2m = -R_w2m C_world.

This reconstruction keeps the released annotation independent of any one depth-rendering run while still producing a complete microphone extrinsic matrix for evaluation.

Citation

If you use SonoWorld or the SonoScene360 dataset, please cite:

@article{jin2026sonoworld,
    title={SonoWorld: From One Image to a 3D Audio-Visual Scene},
    author={Jin, Derong and Chen, Xiyi and Lin, Ming C. and Gao, Ruohan},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2026}
}