TimeGround-1M

Synthetic English audio dataset for time-aware speech understanding, covering temporal localization, temporal description, and timed summaries.

Data Filtering

We use 14k hours of audio from YODAS2 English shards, selected from a 24k-hour source pool after language- and silence-ratio filtering. Synthetic annotations were generated for three time-grounded tasks, then filtered through LLM-based verification, deterministic validity checks, and split-level sampling.

Final rows are grouped by audio duration bucket. train / test values below report unique audio counts, total hours, and median duration.

Tasks

Temporal Localization (TL)

Input: an event description or question. Output: the time interval where it occurs.

Example:

• Q: When does the speaker mention Star Wars midi-chlorians?
• A: Between 01:27:45 and 01:28:09. The speaker compares midi-chlorians to mitochondria.

Temporal Description (TD)

Input: a time interval. Output: a description of what is said in that interval.

Example:

• Q: What issue is raised from 50:20 to 50:35?
• A: Where the line lies between sports journalism and political editorializing, and what role sports media should play in covering civil-rights issues.

Timed Summaries (SUM)

Input: a full audio recording. Output: a segmented summary with segment start timestamps.

Example:

• Prompt: Summarize the English audio with segment start times.
• Output excerpt:
  • 0:08 — Welcome and purpose...
  • 1:26 — Host agenda...
  • 3:17 — USQ student context...

Schema

tl / td

• task_id, audio_id, split, bucket, duration_sec
• audio: path-only HF Audio feature pointing at audio/{split}/.../{id}.wav
• question, answer

sum

• task_id, audio_id, split, bucket, duration_sec
• audio: path-only HF Audio feature
• summary

recordings

One row per long-form recording with nested annotations:

• id, audio, split, bucket, duration_sec, silence_ratio
• tl, td, sum (nested task lists; train sum reflects the same BLEU filter)

Usage

Task rows with streaming:

from datasets import load_dataset

repo_id = "ai-sage/TimeGround-1M"

ds = load_dataset(repo_id, "tl", split="test", streaming=True)
sample = next(iter(ds))

print(sample["question"])
print(sample["answer"])

Decode audio from the task row:

array = sample["audio"]["array"]
sampling_rate = sample["audio"]["sampling_rate"]

Metadata-only (path without audio decoding):

from datasets import Audio, load_dataset

ds = load_dataset(repo_id, "tl", split="test", streaming=True)
ds = ds.cast_column("audio", Audio(decode=False))

sample = next(iter(ds))
print(sample["audio"]["path"])

Full recording with nested annotations:

rec = load_dataset(repo_id, "recordings", split="test", streaming=True)
sample = next(iter(rec))

print(sample["id"], len(sample["tl"]["task_id"]))
print(sample["tl"]["question"][0])
array = sample["audio"]["array"]
sampling_rate = sample["audio"]["sampling_rate"]

Recommended: datasets==3.6.0.

Citation

If you use TimeGround-1M in your research, please cite:

@misc{kutsakov2026_gigachataudio,
  title         = {{GigaChat Audio}: Time-aware Large Audio Language Model},
  author        = {Kutsakov, Aleksandr and
                   Sadovina, Mariia and
                   Gospodinov, Georgii and
                   Maximenko, Alexandr and
                   Kutuzov, Oleg and
                   Bogomolov, Pavel and
                   Minkin, Fyodor},
  year          = {2026},
  eprint        = {2607.10387},
  archivePrefix = {arXiv},
  primaryClass  = {eess.AS},
  url           = {https://arxiv.org/abs/2607.10387},
  note          = {Accepted to Interspeech 2026}
}