This repo contains the automatically annotated dataset, HowToGround1M, introduced in the paper "Large-scale Pre-training for Grounded Video Caption Generation".

📦 Loading the Dataset

You can load each configuration directly with the 🤗 Datasets library:

from datasets import load_dataset

repo = "ekazakos/HowToGround1M"

# Available configs:
# - automatic_annotation_method
# - automatic_annotation_method_keys
# - model_preds
# - model_preds_keys

# automatic_annotation_method: pseudolabels generated using the automatic annotation method proposed in the paper.
ds_auto = load_dataset(repo, "automatic_annotation_method", split="train")

# automatic_annotation_method_keys: contains the corresponding video_ids for the above pseudolabels.
ds_auto_keys = load_dataset(repo, "automatic_annotation_method_keys", split="train")

# model_preds: pseudolabels generated using the trained GROVE model predictions.
ds_preds = load_dataset(repo, "model_preds", split="train")

# model_preds_keys: contains the corresponding video_ids for the above GROVE model predictions.
ds_preds_keys = load_dataset(repo, "model_preds_keys", split="train")

print(ds_auto)
print(ds_preds)

If you use this dataset, please cite:

@inproceedings{kazakos2025grove,
  title     = {Large-scale Pre-training for Grounded Video Caption Generation},
  author    = {Evangelos Kazakos and Cordelia Schmid and Josef Sivic},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year      = {2025}
}