JoyCaption VLM Server

A unified Vision Language Model (VLM) server supporting multiple models for image evaluation, captioning, and analysis. Designed for Linux with AMD GPU (ROCm) support.

Features

• Multi-Model Support: Easy switching between different VLM models
• Dual Backend: Transformers (HuggingFace) and llama.cpp (GGUF models)
• Interactive Chat: Command-line interface for image analysis
• Batch Processing: Evaluate multiple images with rating and filtering
• ROCm Optimized: Full AMD GPU acceleration support
• Flexible Architecture: Simple configuration to switch between 15+ models

Supported Models

Transformers-Based

• JoyCaption: Fast image rating and captioning

GGUF Models (via llama.cpp)

• Mistral-Small 24B: Multiple quantizations (Q8, Q6, Q4)
• MiniCPM-V 4.5: Official and abliterated variants
• Qwen3-VL: 4B, 8B, and 30B parameter models
  • Thinking and Instruct variants
  • Multiple quantizations for different VRAM budgets

System Requirements

• OS: Linux (tested on Ubuntu/Debian)
• GPU: AMD Radeon GPU with ROCm 6.2+ support
• VRAM: 8GB minimum, 24GB+ recommended for larger models
• Python: 3.11+
• CMake: 3.16+ (for building llama.cpp)

Quick Start

1. Install Python with asdf

# Install asdf if not already installed
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.0

# Add to shell profile (bash)
echo '. "$HOME/.asdf/asdf.sh"' >> ~/.bashrc
echo '. "$HOME/.asdf/completions/asdf.bash"' >> ~/.bashrc
source ~/.bashrc

# Install Python plugin
asdf plugin add python

# Install Python 3.11.11
asdf install python 3.11.11

# Set local version
echo "python 3.11.11" > .tool-versions

2. Create Virtual Environment

# Create venv
python -m venv venv

# Activate venv
source venv/bin/activate

3. Install Python Dependencies

# Upgrade pip
pip install --upgrade pip

# Install PyTorch with ROCm 6.2 support
pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm6.2

# Install other dependencies
pip install transformers pillow huggingface_hub accelerate requests

4. Build llama.cpp with ROCm Support

# Clone llama.cpp (if not already present)
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

# Create build directory
mkdir -p build
cd build

# Configure with ROCm support
cmake .. -DGGML_HIPBLAS=ON -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang \
         -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ \
         -DAMDGPU_TARGETS=gfx1100

# Build (use all CPU cores)
cmake --build . --config Release -j$(nproc)

# Verify build
./bin/llama-server --help

Note: Replace gfx1100 with your GPU architecture:

• RX 7900 XTX/XT: gfx1100
• RX 6900 XT: gfx1030
• RX 6800 XT: gfx1030
• Check with: rocminfo | grep gfx

5. Verify GPU Detection

# Check ROCm installation
rocm-smi

# Verify PyTorch ROCm support
python -c "import torch; print('CUDA available:', torch.cuda.is_available()); print('Device:', torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'N/A')"

Expected output:

CUDA available: True
Device: AMD Radeon Graphics

Usage

Interactive Chat Mode

# Start the VLM server in interactive mode
venv/bin/python vlm_server.py --interactive

# Follow prompts to:
# 1. Enter path to image
# 2. Enter your question/prompt about the image
# 3. View model response

Batch Image Evaluation

# Process all images in a directory
venv/bin/python vlm_server.py --mode eval --input-dir ./images

# Images are copied to rated_images/ with ratings in filename
# Example: rate_7_myimage.jpg

API Server Mode (for llama.cpp models)

# Start server
venv/bin/python vlm_server.py --mode server

# Server runs at http://127.0.0.1:8080
# Use --port to change port number

Configuration

Switching Models

Edit vlm_server.py and change the ACTIVE_MODEL variable (around line 174):

# Available options:
ACTIVE_MODEL = "joycaption"                    # Fast rating (Transformers)
ACTIVE_MODEL = "mistral-small-q8"              # 24B model, high quality
ACTIVE_MODEL = "minicpm-v45-abliterated-q8"    # Uncensored 4.5
ACTIVE_MODEL = "qwen3-vl-8b-q8"                # Best 8B model
ACTIVE_MODEL = "qwen3-vl-30b-thinking-q8"      # Largest, most capable

Model Recommendations by VRAM

• 8-12 GB VRAM: qwen3-vl-4b-q4, mistral-small-q4
• 16-20 GB VRAM: qwen3-vl-8b-q8, minicpm-v45-q6
• 24-32 GB VRAM: qwen3-vl-30b-thinking-q8, mistral-small-q8
• 32+ GB VRAM: qwen3-vl-30b-thinking-q8, minicpm-v45-abliterated-f16

Project Structure

.
├── vlm_server.py                      # Main VLM server with multi-model support
├── crawl_and_evaluate_integrated.py   # Batch evaluation pipeline
├── crawl_and_evaluate_server.py       # Server-based evaluation
├── collect_images.py                  # Utility to collect rated images
├── re_evaluate_images.py              # Re-evaluate with different criteria
├── llama.cpp/                         # llama.cpp build (for GGUF models)
│   └── build/bin/llama-server         # Built server binary
├── venv/                              # Python virtual environment
├── images/                            # Input images directory
└── rated_images/                      # Output with rated images
    ├── rate_7_image1.jpg
    ├── rate_7_image1.jpg.log          # Evaluation details
    └── ...

Troubleshooting

GPU Not Detected

# Verify ROCm installation
rocm-smi

# Check ROCm version
apt list --installed | grep rocm

# Reinstall PyTorch with ROCm
pip uninstall torch torchvision
pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm6.2

Out of Memory Errors

1. Switch to smaller quantization: Q8 → Q6 → Q4
2. Use smaller model: 30B → 8B → 4B
3. Reduce batch size in evaluation scripts
4. Close other GPU applications: Check with rocm-smi

llama.cpp Build Fails

# Install build dependencies
sudo apt-get update
sudo apt-get install build-essential cmake git

# Ensure ROCm is installed
sudo apt-get install rocm-hip-sdk

# Verify ROCm compiler
/opt/rocm/llvm/bin/clang++ --version

Model Download Issues

Models are automatically downloaded from HuggingFace Hub:

• Cache location: ~/.cache/huggingface/
• Clear cache: rm -rf ~/.cache/huggingface/hub/models--*
• Check disk space: GGUF models can be 15-30GB each

Server Won't Start

# Check if port is in use
lsof -i :8080

# Kill existing process
pkill -f llama-server

# Check llama-server binary exists
ls -lh llama.cpp/build/bin/llama-server

Performance Notes

Inference Speed (per image)

• JoyCaption: 2-5 seconds
• Qwen3-VL 4B: 3-8 seconds
• Qwen3-VL 8B: 5-15 seconds
• MiniCPM-V 4.5: 10-30 seconds
• Qwen3-VL 30B: 15-45 seconds

VRAM Usage

Advanced Usage

Custom Prompts

Create your own raw_prompts.py:

SYSTEM_PROMPT = "You are an expert image analyst..."

PROMPT = """
Analyze this image and provide:
1. Main subject
2. Visual quality (0-10)
3. Key details
"""

Batch Processing with Filtering

# In crawl_and_evaluate_integrated.py
BATCH_SIZE = 20  # Images per batch
MIN_RATING = 5   # Only keep images rated 5+

License

This project is a collection of scripts for using various VLM models. See individual model licenses:

• JoyCaption: Check fancyfeast/llama-joycaption-beta-one-hf-llava
• Mistral: Check Mistral AI license
• MiniCPM-V: Check openbmb/MiniCPM-V
• Qwen: Check Qwen license
• llama.cpp: MIT License

Contributing

Contributions welcome! Areas for improvement:

• Additional model support
• Web UI interface
• Docker containerization
• Benchmark comparisons
• NVIDIA CUDA support

Support

For issues:

1. Check troubleshooting section above
2. Verify ROCm installation with rocm-smi
3. Check model compatibility with your VRAM
4. Open an issue with system info and error logs