A tool for captioning, visualizing and analyzing image datasets
Python
25
8 commits
updated Oct 23, 2025
This image captioning and visualization tool leverages state-of-the-art generative AI image-to-text models to automatically generate, edit, and manage captions and tags for image datasets. The tool provides a unified interface for accessing multiple vision-language models hosted by leading inference services, including NVIDIA NIM and Hugging Face, enabling users to generate high-quality image descriptions with a single click.
The tool consists of three main modules:
Module 1: Standard Captioning - Generate and edit image captions using various vision-language models (Kosmos-2, Llama 3.2 Vision, Gemma 3, etc.) with customizable parameters and batch processing capabilities.
Module 2: Describe Anything - Advanced localized captioning that allows users to select specific regions within images and generate detailed descriptions for those areas, perfect for datasets requiring precise object or scene descriptions for downstream applications.
Module 2: Data Visualization & Analytics - Comprehensive clustering and analysis tools that help users explore your image-caption datasets through interactive visualizations, keyword filtering, and word cloud generation to identify patterns and themes.
Beyond automated captioning, the tool includes powerful data management features such as bulk editing, find/replace operations, WebDataset export functionality, and interactive dataset exploration capabilities.
Clone the repository with submodules
Option A: Clone with submodules flag
git clone --recurse-submodules <URL>
cd image-captioner
If the dam folder appears empty after cloning:
git submodule update --init --recursive
Set up a virtual environment
Install dependencies
pip install -r requirements.txt
Launch the application
cd scripts
python gradio_interface.py
Access the web interface
For experimentation, a sample image dataset can be downloaded from here. Alternatively, any image or image-text dataset can be used.
Module 2: Tagging via Describe Anything
Module 3: Data Visualization & Analytics
Choose a directory with image samples, and click on the 'Select' button. The selected directory will be visible in the adjacent text box.
Browse through the images using the next and previous buttons.
For each image, the corresponding tag file's contents are read and rendered in the tag text box. If there is no tag file, a new blank text file is created. Users can edit these tags and save them using the button. Alternatively, a new tag can be generated, as described in the next section.
For each image, a new tag can be generated using one of the supported models. Make sure to use a valid API key. Optionally, advanced parameters can be passed to the model as well, if supported. These parameters should be comma separated, and use "=" to assign values.
Sample argument: max_tokens=512, temperature=0.30
Before generating tags, users can choose between the two specified lengths, to prompt the model accordingly.
For a specific type of output (like comma separated tags instead of sentences),
the prompt can be tweaked in model_call.py. This feature is not supported for Hugging Face.
Optionally, strings can be added before/after each tag's text. Please note that these changes are applied to all tags. If a prefix and/or suffix needs to be added to a particular sample only, it can be done by editing it directly.
The loaded dataset can be exported in the WebDataset format. This creates a webdata.zip file which contains all the images along with a json file listing out the image-tag pairs.
Users can also search for specific words/phrases and replace them with the desired keywords.
Advanced parameters:
Advanced parameters: None
To add an image-to-text model available on the currently supported inference services:
model_host_mapping in tagging_utils.py.gen_tag_from_model method, add an elif condition under the corresponding host, along with the model's URL and other required parameters.The Describe Anything tab provides advanced localized captioning capabilities for generating detailed descriptions of specific regions within images. This module extends the core Describe Anything pipeline to support multi-region descriptions and comprehensive caption generation.
This module replicates the core functionality of Module 1 with a new caption generation approach.
Choosing a directory Follow the same steps as in the tagging tab.
Selecting regions for description
Click and drag on the image to select regions of interest. Multiple regions can be selected for comprehensive description generation.
For each selected region, detailed descriptions can be generated using the Describe Anything model. The model provides context-aware captions that describe the specific content within the selected areas.
Generated descriptions can be edited and saved. The system maintains the relationship between image regions and their corresponding descriptions.
After finalizing image-tag pairs, this module can be accessed by clicking on Visualize Data. It provides comprehensive clustering and analysis tools to explore your image-tag datasets through interactive visualizations, keyword filtering, and word cloud generation.
The clustering implementation uses the following components:
This project's MOSAIC - Describe Anything module builds upon the core Describe Anything pipeline from the Describe Anything repository by NVIDIA, UC Berkeley, and UCSF. The original pipeline provides the foundation for detailed localized captioning, which we have extended to support multi-region descriptions and comprehensive caption generation for our image tagging tool.
A tool for captioning, visualizing and analyzing image datasets
Python
25
8 commits
updated Oct 23, 2025
This image captioning and visualization tool leverages state-of-the-art generative AI image-to-text models to automatically generate, edit, and manage captions and tags for image datasets. The tool provides a unified interface for accessing multiple vision-language models hosted by leading inference services, including NVIDIA NIM and Hugging Face, enabling users to generate high-quality image descriptions with a single click.
The tool consists of three main modules:
Module 1: Standard Captioning - Generate and edit image captions using various vision-language models (Kosmos-2, Llama 3.2 Vision, Gemma 3, etc.) with customizable parameters and batch processing capabilities.
Module 2: Describe Anything - Advanced localized captioning that allows users to select specific regions within images and generate detailed descriptions for those areas, perfect for datasets requiring precise object or scene descriptions for downstream applications.
Module 2: Data Visualization & Analytics - Comprehensive clustering and analysis tools that help users explore your image-caption datasets through interactive visualizations, keyword filtering, and word cloud generation to identify patterns and themes.
Beyond automated captioning, the tool includes powerful data management features such as bulk editing, find/replace operations, WebDataset export functionality, and interactive dataset exploration capabilities.
Clone the repository with submodules
Option A: Clone with submodules flag
git clone --recurse-submodules <URL>
cd image-captioner
If the dam folder appears empty after cloning:
git submodule update --init --recursive
Set up a virtual environment
Install dependencies
pip install -r requirements.txt
Launch the application
cd scripts
python gradio_interface.py
Access the web interface
For experimentation, a sample image dataset can be downloaded from here. Alternatively, any image or image-text dataset can be used.
Module 2: Tagging via Describe Anything
Module 3: Data Visualization & Analytics
Choose a directory with image samples, and click on the 'Select' button. The selected directory will be visible in the adjacent text box.
Browse through the images using the next and previous buttons.
For each image, the corresponding tag file's contents are read and rendered in the tag text box. If there is no tag file, a new blank text file is created. Users can edit these tags and save them using the button. Alternatively, a new tag can be generated, as described in the next section.
For each image, a new tag can be generated using one of the supported models. Make sure to use a valid API key. Optionally, advanced parameters can be passed to the model as well, if supported. These parameters should be comma separated, and use "=" to assign values.
Sample argument: max_tokens=512, temperature=0.30
Before generating tags, users can choose between the two specified lengths, to prompt the model accordingly.
For a specific type of output (like comma separated tags instead of sentences),
the prompt can be tweaked in model_call.py. This feature is not supported for Hugging Face.
Optionally, strings can be added before/after each tag's text. Please note that these changes are applied to all tags. If a prefix and/or suffix needs to be added to a particular sample only, it can be done by editing it directly.
The loaded dataset can be exported in the WebDataset format. This creates a webdata.zip file which contains all the images along with a json file listing out the image-tag pairs.
Users can also search for specific words/phrases and replace them with the desired keywords.
Advanced parameters:
Advanced parameters: None
To add an image-to-text model available on the currently supported inference services:
model_host_mapping in tagging_utils.py.gen_tag_from_model method, add an elif condition under the corresponding host, along with the model's URL and other required parameters.The Describe Anything tab provides advanced localized captioning capabilities for generating detailed descriptions of specific regions within images. This module extends the core Describe Anything pipeline to support multi-region descriptions and comprehensive caption generation.
This module replicates the core functionality of Module 1 with a new caption generation approach.
Choosing a directory Follow the same steps as in the tagging tab.
Selecting regions for description
Click and drag on the image to select regions of interest. Multiple regions can be selected for comprehensive description generation.
For each selected region, detailed descriptions can be generated using the Describe Anything model. The model provides context-aware captions that describe the specific content within the selected areas.
Generated descriptions can be edited and saved. The system maintains the relationship between image regions and their corresponding descriptions.
After finalizing image-tag pairs, this module can be accessed by clicking on Visualize Data. It provides comprehensive clustering and analysis tools to explore your image-tag datasets through interactive visualizations, keyword filtering, and word cloud generation.
The clustering implementation uses the following components:
This project's MOSAIC - Describe Anything module builds upon the core Describe Anything pipeline from the Describe Anything repository by NVIDIA, UC Berkeley, and UCSF. The original pipeline provides the foundation for detailed localized captioning, which we have extended to support multi-region descriptions and comprehensive caption generation for our image tagging tool.