The StoryReasoning dataset is a collection of visual storytelling data designed to address limitations in maintaining consistent identity across multiple images while generating coherent narratives. It contains 4,178 cohesive stories derived from 52,016 images, organizing temporally connected image sequences extracted from the same movie scenes to ensure narrative coherence.
Each sample in the dataset contains:
story_id: Unique identifier for each storyimages: Sequence of images (typically 5+ frames) from temporally related scenesframe_count: Number of frames in the storychain_of_thought: Structured scene analysis in text format containing:
story: The generated narrative with XML grounding tags linking text to visual elementsThe stories use four types of specialized XML tags:
<gdi>: Image tags demarcating text segments corresponding to specific frames<gdo>: Entity reference tags linking character and object mentions to their visual counterparts<gda>: Action tags grounding character actions to the individuals performing them<gdl>: Location/landmark tags grounding descriptions to their visual counterpartsThe dataset was created through a systematic process:
To work with the StoryReasoning dataset and its utilities, you can install the story_reasoning package directly from GitHub:
# Install the StoryReasoning package
pip install git+https://github.com/daniel3303/StoryReasoning.git
# Or clone the repository and install locally
git clone https://github.com/daniel3303/StoryReasoning.git
cd StoryReasoning
pip install -e .
After installation, you can import the package and use its components:
from story_reasoning.dataset import StoryReasoningDataset, StoryReasoningAdapter, StoryReasoningUtil
# Now you can use the dataset, adapter, and utilities
This package provides all the necessary tools to work with the dataset, including:
Here's a simple example of loading the dataset:
# Load the dataset directly
from story_reasoning.dataset import StoryReasoningDataset
# Load the complete dataset
story_dataset = StoryReasoningDataset(hf_repo="daniel3303/StoryReasoning")
# Or load just the train split
train_dataset = StoryReasoningDataset(hf_repo="daniel3303/StoryReasoning", split="train")
# Get a sample
sample = train_dataset[0]
The StoryReasoningUtil class provides helpful methods to work with the dataset:
parse_cot() and parse_story()cot_to_string() and story_to_string()strip_story_tags()filter_unmentioned_entities()extract_cot_text() and extract_story_text()The StoryReasoningAdapter helps integrate the dataset with Hugging Face's TRL library for fine-tuning:
from story_reasoning.dataset import StoryReasoningDataset, StoryReasoningAdapter
# Load the dataset
story_dataset = StoryReasoningDataset(hf_repo="daniel3303/StoryReasoning", split="train")
# Create an adapter with options to adjust stories
adapter = StoryReasoningAdapter(
story_dataset,
max_images=10, # Limit the number of images per story
filter_unmentioned_characters=True, # Remove characters not mentioned in the story
filter_unmentioned_objects=True, # Remove objects not mentioned in the story
filter_unmentioned_backgrounds=True # Remove backgrounds not mentioned in the story
)
# Get a TRL-compatible sample
trl_sample = adapter[0]
This dataset can be used for:
If you use this dataset in your research, please cite:
@misc{oliveira2025storyreasoningdatasetusingchainofthought,
title={StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation},
author={Daniel A. P. Oliveira and David Martins de Matos},
year={2025},
eprint={2505.10292},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2505.10292},
}
The dataset is split into training and test sets:
This dataset is licensed under CC-BY-ND-4.0.
For questions or feedback regarding the dataset, please contact:
This dataset builds upon our previous work on GroundCap, which introduced ID-based grounding for single images. While GroundCap demonstrated the effectiveness of structured entity references within individual frames, StoryReasoning extends this approach to sequential contexts, addressing the challenge of maintaining consistent identity across multiple images.
9 commits
1 commits
The StoryReasoning dataset is a collection of visual storytelling data designed to address limitations in maintaining consistent identity across multiple images while generating coherent narratives. It contains 4,178 cohesive stories derived from 52,016 images, organizing temporally connected image sequences extracted from the same movie scenes to ensure narrative coherence.
Each sample in the dataset contains:
story_id: Unique identifier for each storyimages: Sequence of images (typically 5+ frames) from temporally related scenesframe_count: Number of frames in the storychain_of_thought: Structured scene analysis in text format containing:
story: The generated narrative with XML grounding tags linking text to visual elementsThe stories use four types of specialized XML tags:
<gdi>: Image tags demarcating text segments corresponding to specific frames<gdo>: Entity reference tags linking character and object mentions to their visual counterparts<gda>: Action tags grounding character actions to the individuals performing them<gdl>: Location/landmark tags grounding descriptions to their visual counterpartsThe dataset was created through a systematic process:
To work with the StoryReasoning dataset and its utilities, you can install the story_reasoning package directly from GitHub:
# Install the StoryReasoning package
pip install git+https://github.com/daniel3303/StoryReasoning.git
# Or clone the repository and install locally
git clone https://github.com/daniel3303/StoryReasoning.git
cd StoryReasoning
pip install -e .
After installation, you can import the package and use its components:
from story_reasoning.dataset import StoryReasoningDataset, StoryReasoningAdapter, StoryReasoningUtil
# Now you can use the dataset, adapter, and utilities
This package provides all the necessary tools to work with the dataset, including:
Here's a simple example of loading the dataset:
# Load the dataset directly
from story_reasoning.dataset import StoryReasoningDataset
# Load the complete dataset
story_dataset = StoryReasoningDataset(hf_repo="daniel3303/StoryReasoning")
# Or load just the train split
train_dataset = StoryReasoningDataset(hf_repo="daniel3303/StoryReasoning", split="train")
# Get a sample
sample = train_dataset[0]
The StoryReasoningUtil class provides helpful methods to work with the dataset:
parse_cot() and parse_story()cot_to_string() and story_to_string()strip_story_tags()filter_unmentioned_entities()extract_cot_text() and extract_story_text()The StoryReasoningAdapter helps integrate the dataset with Hugging Face's TRL library for fine-tuning:
from story_reasoning.dataset import StoryReasoningDataset, StoryReasoningAdapter
# Load the dataset
story_dataset = StoryReasoningDataset(hf_repo="daniel3303/StoryReasoning", split="train")
# Create an adapter with options to adjust stories
adapter = StoryReasoningAdapter(
story_dataset,
max_images=10, # Limit the number of images per story
filter_unmentioned_characters=True, # Remove characters not mentioned in the story
filter_unmentioned_objects=True, # Remove objects not mentioned in the story
filter_unmentioned_backgrounds=True # Remove backgrounds not mentioned in the story
)
# Get a TRL-compatible sample
trl_sample = adapter[0]
This dataset can be used for:
If you use this dataset in your research, please cite:
@misc{oliveira2025storyreasoningdatasetusingchainofthought,
title={StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation},
author={Daniel A. P. Oliveira and David Martins de Matos},
year={2025},
eprint={2505.10292},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2505.10292},
}
The dataset is split into training and test sets:
This dataset is licensed under CC-BY-ND-4.0.
For questions or feedback regarding the dataset, please contact:
This dataset builds upon our previous work on GroundCap, which introduced ID-based grounding for single images. While GroundCap demonstrated the effectiveness of structured entity references within individual frames, StoryReasoning extends this approach to sequential contexts, addressing the challenge of maintaining consistent identity across multiple images.
9 commits
1 commits