TinyLLaVA

Here, we introduce TinyLLaVA-Qwen2-0.5B-SigLIP , which is trained by the TinyLLaVA Factory codebase. For LLM and vision tower, we choose Qwen2-0.5B and siglip-so400m-patch14-384, respectively.

Usage

Execute the following test code:

from tinyllava.eval.run_tiny_llava import eval_model

model_path = 'Zhang199/TinyLLaVA-Qwen2-0.5B-SigLIP'
prompt = "What are the things I should be cautious about when I visit here?"
image_file = "https://llava-vl.github.io/static/images/view.jpg"
conv_mode = "qwen2_base" # or llama, gemma, etc

args = type('Args', (), {
    "model_path": model_path,
    "model": None,
    "query": prompt,
    "conv_mode": conv_mode,
    "image_file": image_file,
    "sep": ",",
    "temperature": 0,
    "top_p": None,
    "num_beams": 1,
    "max_new_tokens": 512
})()

eval_model(args)

Result

P.S. TinyLLaVA Factory is an open-source modular codebase for small-scale LMMs with a focus on simplicity of code implementations, extensibility of new features, and reproducibility of training results. This code repository provides standard training&evaluating pipelines, flexible data preprocessing&model configurations, and easily extensible architectures. Users can customize their own LMMs with minimal coding effort and less coding mistake.

TinyLLaVA Factory integrates a suite of cutting-edge models and methods.

• LLM currently supports OpenELM, TinyLlama, StableLM, Qwen, Gemma, Phi, and Qwen2.
• Vision tower currently supports CLIP, SigLIP, Dino, and combination of CLIP and Dino.
• Connector currently supports MLP, Qformer, and Resampler.