Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference

Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference by Siyuan Wang, Dianyi Wang, Chengxing Zhou, Zejun Li, Zhihao Fan, Xuanjing Huang, and Zhongyu Wei.

Abstract. Large Vision-Language Models (LVLMs) typically learn visual capacity through visual instruction tuning, involving updates to both a projector and their LLM backbones. Inspired by the concept of a visual region in the human brain, we investigate the existence of an analogous visual region within LLMs that functions as a cognitive core, and explore the potential of efficient training of LVLMs via selective layers tuning. Using Bunny-Llama-3-8B-V for detailed analysis and other three LVLMs for validation across diverse visual and textual tasks, we find that selectively updating 25% of LLMs layers, when sparsely and uniformly distributed, can preserve nearly 99% of visual performance and maintain or improve textual task results, while effectively reducing training time. Based on this targeted training approach, we further propose a novel visual region-based pruning paradigm, removing non-critical layers outside the visual region, which can achieve minimal performance loss. This study offers an effective and efficient strategy for LVLM training and inference by activating a layer-wise visual region within LLMs, which proves consistently effective across different models.

Release

• [2025/05/15] 🔥🔥🔥 Our paper has been accepted by ACL 2025! 🔥🔥🔥
• [2024/12/17] 🔥 we release our paper, checkout the paper for details.

Contents

• Install
• Train
• Evaluation

Install

• CUDA and cuDNN

  We use CUDA 11.8 and cuDNN 8.7.0. We actually use the CUDA docker by NVIDIA: docker pull nvcr.io/nvidia/cuda:11.8.0-cudnn8-devel-ubuntu20.04. CUDA 12 is fine, too.

• Create a conda virtual environment and activate it:

  conda create -n bunny python=3.10
  conda activate bunny
  

  pip install --upgrade pip  # enable PEP 660 support
  

• Install apex

  # https://github.com/NVIDIA/apex#from-source
  pip install ninja
  git clone https://github.com/NVIDIA/apex
  cd apex
  # if pip >= 23.1 (ref: https://pip.pypa.io/en/stable/news/#v23-1) which supports multiple `--config-settings` with the same key...
  pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
  # otherwise
  pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./
  

• Install flash-attention

  # https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#installation-and-features
  pip install packaging
  pip install flash-attn --no-build-isolation
  

• Install bunny and other requirements

  cd Bunny
  pip install -e .
  

Training

Our model is trained on 8 A100 GPUs. Under other circumstances, you can reduce the per_device_train_batch_size and increase the gradient_accumulation_steps accordingly. Always keep the global batch size the same: global_batch_size = per_device_train_batch_size $\times$ gradient_accumulation_steps $\times$ num_gpus.

• Experiments model components

the "siglip-so400m-patch14-384" is for Bunny-Llama-3-8B-V and Bunny-Phi3-mini-4B-V, the "clip-336" is for LLaVA-1.5-7B/13B

Activating visual region Finetuning

we focus on the supervised fine-tuning stage using 695K and 665K language-image instruction-following instances for Bunny and LLaVA, and We have inherited the corresponding LVLM pre-training connector's checkpoints. Considering computational constraints, we use LoRA ,highlighting that our Activating visual region Finetuning is complementary to other efficient training methods.

The training script for activating visual region stage with DeepSpeed ZeRO-3 can be found in scripts/train/finetune_visual_region.sh. Global Batch Size is 128 we utilize peft lora, the lora_r is 128 and lora_alpha is 256.

• Run

  Update --model_name_or_path and --vision_tower to the paths of the base_llm_model and vision encoder, respectively. Update MODEL_TYPE, PRETRAIN_DIR(to ref Pretrained_CKPT) and OUTPUT_DIR accordingly.

For the --layer_selection, you can choose the layers corresponding to the visual area for training. In our experiments, we selected eight layers for Bunny-Llama-3-8B-V, Bunny-Phi3-mini-4B-V, and LLaVA-1.5-7B, while for LLaVA-1.5-13B, we selected nine layers. Additionally, the Sparse & Uniform selection method yields the best results. For more details, please refer to the paper for further adjustments to the training layers.

Evaluation

see evaluation_full.md.

Acknowledgement

• LLaVA: the dataset we utilized.
• Bunny: the codebase we built upon and the dataset we utilized.