ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation

Github | Dataset(ModelScope) | Model | Paper

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in chart understanding tasks. However, interpreting charts with textual descriptions often leads to information loss, as it fails to fully capture the dense information embedded in charts. In contrast, parsing charts into code provides lossless representations that can effectively contain all critical details. Although existing open-source MLLMs have achieved success in chart understanding tasks, they still face two major challenges when applied to chart-to-code tasks: (1) Low executability and poor restoration of chart details in the generated code and (2) Lack of large-scale and diverse training data. To address these challenges, we propose ChartCoder, the first dedicated chart-to-code MLLM, which leverages Code LLMs as the language backbone to enhance the executability of the generated code. Furthermore, we introduce Chart2Code-160k, the first large-scale and diverse dataset for chart-to-code generation, and propose the Snippet-of-Thought (SoT) method, which transforms direct chart-to-code generation data into step-by-step generation. Experiments demonstrate that ChartCoder, with only 7B parameters, surpasses existing open-source MLLMs on chart-to-code benchmarks, achieving superior chart restoration and code excitability. Our code is available at this https URL .

This repository contains the code to train and infer ChartCoder.

Overview

News

• [2025.5.17] ChartCoder has been accepted by ACL 2025 Main.
• [2025.3.13] We have upload our dataset Chart2Code-160k(HF) to Huggingface.
• [2025.2.19] We have released our dataset Chart2Code-160k to ModelScope.
• [2025.1.16] We have updated our data generation code data_generator, built on Multi-modal-Self-instruct. Please follow their instructions and our code to generate the <chart, code> data pairs.

Installation

1. Clone this repo

git clone https://github.com/thunlp/ChartCoder.git

2. Create environment

cd ChartCoder
conda create -n chartcoder python=3.10 -y
conda activate chartcoder
pip install --upgrade pip # enable PEP 660 support
pip install -e .

3. Additional packages required for training

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Models

The MLP Connector is our pre-trained MLP weights, which you could directly use for SFT.

Data

Train

The whole training process consists of two stages. To train the ChartCoder, siglip-so400m-patch14-384 and deepseek-coder-6.7b-instruct should be downloaded first.

For Pre-training, run

bash scripts/train/pretrain_siglip.sh

For SFT, run

bash scripts/train/finetune_siglip_a4.sh

Please change the model path to your local path. See the corresponding .sh file for details. We also provide other training scripts, such as using CLIP _clip and multiple machines _m. See scripts/train for further information.

Inference

Please see inference.py for details.

Results

Please refer to our paper for detailed performance on ChartMimic, Plot2Code and ChartX benchmarks. Thanks for these contributions to the chart-to-code field.

Contact

For any questions, you can contact 2429527z@gmail.com.

Citation

If you find this work useful, consider giving this repository a star ⭐️ and citing 📝 our paper as follows:

@misc{zhao2025chartcoderadvancingmultimodallarge,
      title={ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation},
      author={Xuanle Zhao and Xianzhen Luo and Qi Shi and Chi Chen and Shuo Wang and Wanxiang Che and Zhiyuan Liu and Maosong Sun},
      year={2025},
      eprint={2501.06598},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2501.06598},
}

Acknowledgement

The code is based on the LLaVA-NeXT. Thanks for these great works and open sourcing!