FT-LLM 2026: COMPASS — Japanese Vision-Language Model Fine-tuning Pipeline

Japanese README / 日本語版 README はこちら

This project was conducted as part of the free-form task of FT-LLM 2026.

Title: Development of a Japanese Financial VLM through Integration of Reasoning Enhancement and Document Comprehension (推論強化と文書読解の統合による日本語金融VLMの開発)

This research was carried out by Atsushi Yanagisawa and Genshin Kakimoto.

📄 Publication

This work has been accepted at the ICML 2026 Workshop on Continual Adaptation at Scale: Towards Sustainable AI (CATS).

Staged Continual Adaptation of Multimodal Foundation Models for Japanese Financial Documents Genshin Kakimoto, Atsushi Yanagisawa (Kyoto University)

For training results and detailed research findings, please refer to the blog posts below.

• HuggingFace Collection
• Blog (English)
• Blog (日本語)

Model Architecture

graph TD
    A["Input Image"]
    B["Input Text"]
    A --> C["SigLIP-v2 Vision Encoder"]
    C --> D["MLP Projector"]
    D --> E["LLM-JP-4-8B-Instruct Large Language Model"]
    B --> E
    E --> F["Output Text"]

Pipeline

The pipeline execution order follows the numerical sequence:

phase1 (VLM pre-training)
  ├── Stage 1-1: Image Caption Pretraining (projector only)
  └── Stage 1-2: Visual Instruction Tuning (projector + LLM)
        └─→ phase2 (LLM fine-tuning)
              ├── SFT: Knowledge distillation from Qwen3-30B
              └── DPO: Direct Preference Optimization
                    └─→ phase3 (Domain fine-tuning)
                          ├── TAT-QA / ConvFinQA / FinQA
                          └── Domain-specific QA
                                └─→ eval (Evaluation)
                                      ├── GSM8K (Math)
                                      ├── JP Harness (Financial MCQ)
                                      └── EDINET Bench (Classification)

Note: The scraping and data generation scripts (scraping/, data-prepare/) may have Python syntax issues. These scripts are provided as reference only.

Directory Structure

Key Datasets

Training Datasets

Evaluation Benchmarks

Quick Start

Environment

• Python 3.10+
• CUDA 12.1+
• 1+ NVIDIA GPU (H100 80GB recommended for full pipeline)

python -m venv venv
source venv/bin/activate    # Linux/Mac
venv\Scripts\activate       # Windows
pip install -r phase1/requirements.txt       # VLM pre-training
pip install -r phase3/requirements.txt       # Domain fine-tuning

Running the Pipeline (Model B)

# Set environment variables
export LLM_PATH="/path/to/llm-jp-4-8b-instruct"
export VG_IMAGE_DIR="/path/to/visual_genome"
export HF_TOKEN="hf_xxxxx"

# Phase 1 Stage 1-1: Caption + OCR Pretraining (Projector only)
python phase1/train_stage1_1.py \
    --llm_path $LLM_PATH \
    --vision_tower google/siglip2-so400m-patch14-384 \
    --no_anyres \
    --output_dir ./outputs/modelB/phase1/stage1-1 \
    --ocr_mode huggingface \
    --batch_size 2 \
    --gradient_accumulation_steps 64 \
    --epochs 2 \
    --learning_rate 1e-3 \
    --warmup_ratio 0.03 \
    --max_seq_length 2048 \
    --mixed_precision bf16 \
    --save_steps 500 \
    --logging_steps 10

# Phase 1 Stage 1-2: Instruction + QA Tuning (VE + Projector + LLM)
python phase1/train_stage1_2.py \
    --llm_path ./outputs/modelB/phase1/stage1-1/checkpoint-best \
    --base_llm_path $LLM_PATH \
    --vision_tower google/siglip2-so400m-patch14-384 \
    --use_anyres \
    --vision_lr 2e-6 \
    --output_dir ./outputs/modelB/phase1/stage1-2 \
    --use_huggingface \
    --image_dir $VG_IMAGE_DIR \
    --batch_size 2 \
    --gradient_accumulation_steps 128 \
    --epochs 1 \
    --learning_rate 1e-5 \
    --warmup_ratio 0.03 \
    --max_seq_length 2048 \
    --mixed_precision bf16 \
    --use_lora \
    --lora_r 64 \
    --lora_alpha 128 \
    --save_steps 500 \
    --logging_steps 10

# LoRA Merge (Phase 1)
python phase2/merge_adapter.py \
    --base_model $LLM_PATH \
    --adapter_path ./outputs/modelB/phase1/stage1-2/checkpoint-best \
    --output_dir ./outputs/modelB/phase1/merged_llm \
    --torch_dtype bfloat16

cp ./outputs/modelB/phase1/stage1-2/checkpoint-best/mm_projector.bin \
   ./outputs/modelB/phase1/merged_llm/

# Phase 2 SFT
python phase2/SFT/train_sft.py \
    --base_model ./outputs/modelB/phase1/merged_llm \
    --hf_dataset Yana/ft-llm-2026-reasoning-sft \
    --output_dir ./outputs/modelB/phase2/SFT \
    --dataset_fraction 1.0 \
    --learning_rate 2e-4 \
    --global_batch_size 64 \
    --micro_batch_size 2 \
    --num_epochs 1 \
    --lora_r 32 \
    --lora_alpha 64 \
    --max_seq_length 2048 \
    --num_gpus 1 \
    --gradient_checkpointing \
    --warmup_ratio 0.03 \
    --save_steps 500

# Phase 2 Merge
python phase2/merge_adapter.py \
    --base_model ./outputs/modelB/phase1/merged_llm \
    --adapter_path ./outputs/modelB/phase2/SFT \
    --output_dir ./outputs/modelB/phase2/merged \
    --torch_dtype bfloat16

# Phase 3: Domain Fine-tuning (example: ConvFinQA)
python phase3/download_datasets.py --output_dir ./datasets --datasets all

python phase3/train_convfinqa.py \
    --llm_path ./outputs/modelB/phase2/merged \
    --data_path ./datasets/ConvFinQA/data.zip \
    --output_dir ./outputs/modelB/phase3/convfinqa \
    --data_format turn \
    --batch_size 2 \
    --gradient_accumulation_steps 8 \
    --epochs 3 \
    --learning_rate 5e-6 \
    --max_seq_length 2048 \
    --mixed_precision bf16 \
    --use_lora \
    --lora_r 16 \
    --lora_alpha 32 \
    --save_steps 500 \
    --logging_steps 10

# Evaluation
python eval/gsm8k/eval_gsm8k.py \
    --model_path ./outputs/modelB/phase2/merged \
    --output_dir ./eval_results/gsm8k \
    --n_shots 8 \
    --torch_dtype bf16

Documentation

• Phase 1: VLM Pre-training
• Phase 2: LLM Fine-tuning
• Phase 3: Domain Fine-tuning
• Evaluation
• VLM Build Guide (Full pipeline details)

Notes

• Checkpoint/Resume: All pipelines support checkpoint-based resume. Re-running the same command automatically resumes from the last checkpoint.
• Dataset Caching: Training scripts cache tokenized data as .pkl files. Delete cache if the dataset changes.
• Scraping: Scraping scripts have built-in delays (0.5-1.0s). Do not reduce to avoid server overload. There may be Python syntax issues in these scripts.
• Development Environment: Use python (not python3) as the primary command (Windows development environment).

Contributions

Both authors jointly decided the overall research direction. Genshin Kakimoto led the Phase 2 dataset creation and fine-tuning; Atsushi Yanagisawa led all other experiments (Phase 1 & 3 datasets and fine-tuning), evaluation, and paper writing.

License

This project is licensed under the MIT License.