Salesforce/CogAlign

Dataset

Dataset Card for CogAlign

6

14 commits

1 linked in READMEs

updated Jun 28, 2025

See the code

README

Dataset Card for CogAlign

Dataset Description

CogAlign is a post-training strategy for Vision Language Models (VLMs) aimed at enhancing their visual arithmetic capabilities. This repository presents the training data for CogAlign, a synthetic dataset containing 64,000 examples designed to facilitate this post-training process.

CogAlign is inspired by Piaget's theory of cognitive development and focuses on improving a VLM's understanding of conservation and decentration. Each example includes a visual input, a query prompting comparison of a specific property, a positive response consistent with the visual input, and a negative response that contradicts it.

Training VLMs with CogAlign leads to performance improvements in downstream tasks that rely on visual arithmetic, specifically:

  • Chart Understanding: When used to train VLMs, CogAlign leads to an average performance increase of 4.6% on the CHOCOLATE chart understanding dataset.

  • Geometric Problem-Solving: Models trained with CogAlign exhibit an average performance gain of 2.9% on the subset of MATH-VISION dataset that focuses on geometry-related questions.

This dataset allows VLMs to learn fundamental visual arithmetic, leading to better performance in tasks involving visual arithmetic. Importantly, CogAlign has been shown to achieve comparable or even better performance than task-specific SFT methods, while requiring significantly less (60%) training data.

Citation

If you find CogAlign useful in your research, please consider citing:

@misc{huang-etal-2025-cogalign,
    title = "Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding",
    author = "Huang, Kung-Hsiang  and
      Qin, Can  and
      Qiu, Haoyi  and
      Laban, Philippe  and
      Joty, Shafiq  and
      Xiong, Caiming  and
      Wu, Chien-Sheng",
    year = "2025",
    archivePrefix = "arXiv",
    primaryClass={cs.AI}
}

Contributors

khhuang

14 commits

Salesforce/CogAlign

Dataset

Dataset Card for CogAlign

6

14 commits

1 linked in READMEs

updated Jun 28, 2025

See the code

README

Dataset Card for CogAlign

Dataset Description

CogAlign is a post-training strategy for Vision Language Models (VLMs) aimed at enhancing their visual arithmetic capabilities. This repository presents the training data for CogAlign, a synthetic dataset containing 64,000 examples designed to facilitate this post-training process.

CogAlign is inspired by Piaget's theory of cognitive development and focuses on improving a VLM's understanding of conservation and decentration. Each example includes a visual input, a query prompting comparison of a specific property, a positive response consistent with the visual input, and a negative response that contradicts it.

Training VLMs with CogAlign leads to performance improvements in downstream tasks that rely on visual arithmetic, specifically:

  • Chart Understanding: When used to train VLMs, CogAlign leads to an average performance increase of 4.6% on the CHOCOLATE chart understanding dataset.

  • Geometric Problem-Solving: Models trained with CogAlign exhibit an average performance gain of 2.9% on the subset of MATH-VISION dataset that focuses on geometry-related questions.

This dataset allows VLMs to learn fundamental visual arithmetic, leading to better performance in tasks involving visual arithmetic. Importantly, CogAlign has been shown to achieve comparable or even better performance than task-specific SFT methods, while requiring significantly less (60%) training data.

Citation

If you find CogAlign useful in your research, please consider citing:

@misc{huang-etal-2025-cogalign,
    title = "Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding",
    author = "Huang, Kung-Hsiang  and
      Qin, Can  and
      Qiu, Haoyi  and
      Laban, Philippe  and
      Joty, Shafiq  and
      Xiong, Caiming  and
      Wu, Chien-Sheng",
    year = "2025",
    archivePrefix = "arXiv",
    primaryClass={cs.AI}
}

Contributors

khhuang

14 commits