single-image

cargo run  # default args, use liuhaotian/llava-v1.6-vicuna-7b, default-image is image/llava_logo.png, prompt is "is this a cat?"
cargo run  -- --image-file "images/llava_v1_5_radar.jpg" --prompt "what does this picture show?"
cargo run -- --model-path "fancyfeast/llama-joycaption-beta-one-hf-llava" 

task

• Download the corresponding weights from Hugging Face

• Load the model weights and configs

  • general llava config(need to rethink what is necessary)
  • Vision tower(CLIP)
    • image processor(partial, the format of 'size' and 'crop size' not fully compatible with python transformer)
  • LLM
    • llama/vicuna
    • mistral

• image preprocess

  • clip image processor
  • 'anyres' image preprocess
  • 'pad' image preprocess

• conv template (partial, only implement conv_llava_v1 and conv_chatml_direct, which is enough for LLaVA v1.6)

• Model structure Implementation

  • Vision tower
  • LLM
    • modify of llama code
      • output embedding result
      • generate from embed tensors

• model forward

  • Vision tower
    • feature select
  • LLM
  • process of multiple images
    • read multiple images
    • multiple images patch process
  • concat of image features and text features
  • truncate of the concat features

• main process

  • load model
  • load image
  • load text
  • tokenize text
  • forward
    • single image
  • output
  • KV cache
  • conversation mode

• quantization

  • 4-bit
  • 8-bit

• top priority migrate to support llava-hf series model

  • determine whether it is a llava-hf model
  • translate of config
  • translate of model
  • take care of constant such as image_token_index
  • modify of image processor config

Tokenizer Setup

conda create -n llava python=3.10  
pip install transformers protobuf