CueFlow: Context-Guided Causal Flow Adaptation for Chart Question Answering in DeepSeek-OCR2

Environment

Refer to https://huggingface.co/deepseek-ai/DeepSeek-OCR-2 or our requirements.txt

Training

Download checkpoint from https://huggingface.co/deepseek-ai/DeepSeek-OCR-2

Choose an architecture and replace .py with our modelings//*.py

cd ./ms-swift

DEEPSEEK_OCR2_ENABLE_TEXT_QUERY_RESIDUAL=1 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
MPLCONFIGDIR=/tmp/matplotlib \
swift sft \
  --use_hf 1 \
  --model /home/jiaqiliu/scratch/532G/ms-swift/output/selfattnproj_lora/v3-20260411-040536/checkpoint-3537 \
  --dataset HuggingFaceM4/ChartQA \
  --tuner_type full \
  --torch_dtype float16 \
  --ddp_find_unused_parameters true \
  --num_train_epochs 1 \
  --per_device_train_batch_size 4 \
  --per_device_eval_batch_size 4 \
  --gradient_accumulation_steps 2 \
  --learning_rate 1e-4 \
  --lr_scheduler_type cosine_with_min_lr \
  --lr_scheduler_kwargs '{"min_lr": 1e-5}' \
  --warmup_ratio 0.02 \
  --weight_decay 0.0 \
  --freeze_llm true \
  --freeze_vit true \
  --freeze_aligner true \
  --freeze_parameters_ratio 1 \
  --trainable_parameters model.qwen2_model.text_context_proj model.qwen2_model.text_query_mlp model.qwen2_model.model
  --gradient_checkpointing false \
  --max_pixels 602112 \
  --max_length 2048 \
  --dataloader_num_workers 8 \
  --dataset_num_proc 8 \
  --lazy_tokenize true \
  --logging_steps 5 \
  --output_dir output/cueflow

Evaluation

Using trained model or download from https://huggingface.co/JiaqiLiu/CueFlow

CueFlow/DeepSeek-OCR-2

python run_deepseek_ocr2_eval_vllm_textquery.py --datasets ChartQA ChartQAPro --split test --model-name $model_path --text-query-residual --output-dir $output_path

DeepSeek-OCR

python3 run_deepseek_ocr1_eval_vllm.py --datasets ChartQA --split test --model-name $moded_path --output-dir $output_path

Visualization

CueFlow/DeepSeek-OCR-2

python /home/jiaqiliu/scratch/532G/Deepseek-OCR2-ChartQA/run_deepseek_ocr2_eval_vllm_vis_eachquery_middle.py \
  --datasets ChartQA \
  --split test \
  --model-name $model_path \
  --num-cases 128 \
  --deepseek-vllm-dir ./DeepSeek-OCR-2/DeepSeek-OCR2-master/DeepSeek-OCR2-vllm \
  --output-dir $vis_path

DeepSeek-OCR-1

python /home/jiaqiliu/scratch/532G/Deepseek-OCR2-ChartQA/run_deepseek_ocr1_clip_attention_vis_dataset.py \
  --datasets ChartQA \
  --split test \
  --num-cases 16 \
  --seed 42 \
  --model-path $model_path \
  --output-dir $vis_path \
  --branch global \
  --layers $clip_layer \
  --token-mode patch \
  --token-row 8 \
  --token-col 8 \
  --dtype bfloat16