VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning

This repository contains the datasets used and generated in the paper VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning.

The work introduces VinciCoder, a unified multimodal code generation model that addresses the limitations of single-task training paradigms. It proposes a two-stage training framework, beginning with a large-scale Supervised Finetuning (SFT) corpus of 1.6M image-code pairs for direct code generation and visual-based code refinement. Subsequently, a Visual Reinforcement Learning (ViRL) strategy is introduced, employing a coarse-to-fine reward mechanism to improve visual fidelity.

Paper: https://huggingface.co/papers/2511.00391 Code / Project Page: https://github.com/DocTron-hub/VinciCoder

Dataset Structure

The VinciCoder project utilizes two main types of datasets:

SFT Dataset

The Supervised Finetuning (SFT) dataset comprises 1.6M image-code pairs. This dataset is a compilation from various works in chart-to-code, web-to-HTML, image-to-SVG, image-to-Latex, and other domains. It also includes optimized and newly generated data.

The SFT dataset is available at VinciCoder_SFT_Data.

RL Dataset

The Reinforcement Learning (RL) dataset contains 42k data points sourced from five domains, used for the Visual Reinforcement Learning (ViRL) stage of VinciCoder.

The RL dataset is available at VinciCoder_RL_Data.

Citation

If you find this work useful, please consider giving this repository a star ⭐️ and citing 📝 our paper as follows:

@misc{zhao2025vincicoderunifyingmultimodalcode,
      title={VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning}, 
      author={Xuanle Zhao and Deyang Jiang and Zhixiong Zeng and Lei Chen and Haibo Qiu and Jing Huang and Yufeng Zhong and Liming Zheng and Yilin Cao and Lin Ma},
      year={2025},
      eprint={2511.00391},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.00391}, 
}

Acknowledgement

The training frameworks are based on ms-swift and EasyR1. Thanks for these great works and open sourcing!