---

license: mit language:

• en

---

Annotation Overview

We design Charades-CON and ActivityNet-CON to evaluate the consistency of Video-LLMs in temporal comprehension based on two widely-used temporal grounding benchmarks: Charades-STA and ActivityNet-Captions.

Example of Annotations

{
  "WIOOY": {
    "duration": 30.7,
    "sentences": ["person puts bag of groceries on table."],
    "timestamps": [[2.9, 9.1]],
    "consistency_annotations": [
      {
        "A": [
          "The person places a bag of groceries onto the table.",
          "A bag of groceries is put on the table by the person.",
          "On the table, the person puts a bag of groceries."
        ],
        "M": [
          "A person throws a bag of groceries on the floor.",
          "Person puts books on the table.",
          "Someone takes a bag of groceries from the table."
        ],
        "C": {
          "Y": [
            "Does the person put a bag of groceries on the table?",
            "Is there a bag of groceries being placed?"
          ],
          "N": [
            "Does the person put a bag of groceries on the chair?",
            "Is the table empty?"
          ]
        }
      }
    ],
    "shifted_timestamps": [[13, 19.2]]
  }
}

Key Components of the Annotations

• Key: Represents a unique video ID (e.g., WIOOY).
• Duration: The total length of the video in seconds.
• Sentences: A natural language description of the action occurring in the video.
• Timestamps: The time intervals during which the action occurs.

Consistency Annotations

Aligned Sentences (A): Variants of the original sentence that convey the same meaning.

Misaligned Sentences (M): Sentences that describe different meanings from the original sentence.

Compositional Queries (C): Questions that test understanding of parts of the sentence:

• Y (Yes): Questions that are expected to have a positive answer 'Yes'.

• N (No): Questions that are expected to have a negative answer 'No'.

Shifted Timestamps: We randomly moved the timestamp to another segment in the video to generate this.

Licensing Information

The content of the original dataset adheres to its respective license.

For tasks involving datasets with unknown or custom licenses, we recommend that users refer to the original project or reach out to the dataset owner for detailed licensing information.

Paper and Code for more information: Paper, Code

Citation

If you find our research and codes useful, please consider starring our repository and citing our paper:

@article{jung2024consistency,
  title={On the Consistency of Video Large Language Models in Temporal Comprehension},
  author={Jung, Minjoon and Xiao, Junbin and Zhang, Byoung-Tak and Yao, Angela},
  journal={arXiv preprint arXiv:2411.12951},
  year={2024}
}