Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models

Hulingxiao He · Geng Li · Zijun Geng · Jinglin Xu · Yuxin Peng

ICLR 2025

Paper | Model | Poster | Slides | Video

TL;DR: We revisit three quintessential capabilities of MLLMs for FGVR, including object information extraction, category knowledge reserve, object-category alignment, and position of the root cause as a misalignment problem. To address this issue, we present Finedefics, an MLLM that enhances the model's FGVR capability by incorporating informative attribute descriptions of objects into the training phase.

📣 News

• [04/21/2025] We release the training code and data of Finedefics! Welcome to follow our work and give us a star 🌟!
• [04/21/2025] We release the poster, slides, and video of Findefics.
• [02/12/2025] We release the model Finedefics and evaluation code.
• [01/23/2025] Our work is accepted to ICLR 2025 🌼! See you in Singapore this April!

🧠 More Related Research

• TARA (CVPR 2026): using fine-grained category tree to boost hierarchical visual recognition capability of MLLMs. 【Paper】【Code】
• Fine-R1 (ICLR 2026): the first MLLM to surpass various strong CLIP-like models (e.g., SigLIP-L) in FGVR. 【Paper】【Code】

💾 Installation

1. Clone this repository and move to the project working directory:

git clone https://github.com/PKU-ICST-MIPL/Finedefics_ICLR2025.git
cd Finedefics_ICLR2025

2. Install the working environment step-by-step

conda create -n finedefics python=3.10.14 -y # create finedefics conda environment
conda activate finedefics

pip install torch==2.4.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
cd transformers
pip install -e .

📦 Datasets Preparation

We can download prepared FGVR datasets here:

• Baidu Yun Pan: ibje

After downloading, unzip and organize the directory like the following and we are ready to go:

Finedefics_ICLR2025
    └── data
          ├── aircraft-102 ✈️
          ├── bird-200 🦤
          ├── car-196 🚙
          ├── dog-120 🐕
          ├── flower-102 🌼
          ├── pet-37 🐈
          ├── pretrain.csv
          └── finetune.csv

⚗ Training

1. Download the base model Idefics2 from HuggingFace:

huggingface-cli download HuggingFaceM4/idefics2-8b --local-dir /path/to/save/model

After downloading, organize the directory like the following:

Finedefics_ICLR2025
    └── pretrained_weights
          └── idefics2-8b

1. We then train Finedefics using training sets of all 6 FGVR datasets.

sh train.sh

📋 Evaluation

We use FOCI-Benchmark to evaluate our model.

1. Preparing the Environment

Requirements can be found in requirements.txt. We recommend using Python ≥ 3.9 and PyTorch ≥ 2.2.1.

2. Running the Benchmark

An example of evaluating on dog-120 dataset is:

python run_ic_bench.py --model=/path/to/model --dataset=dog-120 --prompt_query='Which of these dogs is shown in the image?' --image_root=/path/to/dog-120 --batchsize=4

Note: Available datasets are aircraft-102, bird-200, car-196, dog-120, flower-102, pet-37.

See scripts for examples of evaluating Finedefics on all benchmark datasets.

3. Testing New Models

Our code is trivial to extend to new models, especially if they use HuggingFace:

• Implement the model based on the reference HfModel or the other implemented models.
• Update model_template() to provide the model instruction template.
• Update load_model() to load the model based on the name.

4. Testing on New Datasets

Our code is also trivial to extend to new image classification datasets:

• Implement a loader function that creates a dictionary mapping labels to (relative) image paths and add it to DATASET_TO_LOADER.
• When running the benchmark for the first time, we use CLIP to find difficult multiple-choice options and store them in data for subsequent runs.

🚩 Acknowledgments

Our code references FineR, FOCI-Benchmark, HACL. Many thanks to the authors.

📝 Citation

If you find it useful for your research and applications, please cite related papers using this BibTeX:

@article{he2025analyzing,
  title={Analyzing and boosting the power of fine-grained visual recognition for multi-modal large language models},
  author={He, Hulingxiao and Li, Geng and Geng, Zijun and Xu, Jinglin and Peng, Yuxin},
  journal={arXiv preprint arXiv:2501.15140},
  year={2025}
}

@article{he2026fine,
  title={Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning},
  author={He, Hulingxiao and Geng, Zijun and Peng, Yuxin},
  journal={arXiv preprint arXiv:2602.07605},
  year={2026}
}

@article{he2026taxonomy,
  title={Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models},
  author={He, Hulingxiao and Tan, Zhi and Peng, Yuxin},
  journal={arXiv preprint arXiv:2603.00431},
  year={2026}
}