Overview

AutoVision-Annotator combines traditional computer vision models with state-of-the-art Vision-Language Models (VLMs) to provide a flexible solution for object detection, segmentation, and annotation tasks. Objects are detected dynamically from natural language prompts, results are exported to Label Studio-compatible formats, and validated annotations feed back into model training, forming a complete annotation loop.

Architecture

flowchart LR
    subgraph IF["User Interfaces"]
        FE["React Frontend<br/>(Vite)"]
        GA["Gradio App<br/>tools/gradio_vlm.py"]
    end

    subgraph BE["FastAPI Backend"]
        EP["REST Endpoints<br/>/api/vlm"]
        PM["Pipeline Manager<br/>(singleton)"]
    end

    subgraph CORE["AutoVisionAnnotatorModels"]
        direction TB
        DP["Qwen Object Detection Pipeline<br/>VLM-only / Hybrid Parallel / Hybrid Sequential"]
        VB["VLM Backends<br/>HuggingFace / vLLM / Ollama<br/>Qwen2.5-VL, BLIP-2, LLaVA, SmolVLM, GLM-4.5V"]
        TD["Traditional Detectors<br/>YOLO / DETR / RT-DETR"]
        SM["SAM Segmentation"]
        FUS["Fusion and Matching<br/>WBF / NMS / IoU Overlap"]
        VP["Video Pipeline"]
    end

    subgraph LOOP["Annotation Loop"]
        EXP["Label Studio JSON Export"]
        GCS[("Google Cloud Storage")]
        HU["Human Review<br/>(Label Studio)"]
        TRN["Model Fine-Tuning<br/>(HF Trainer)"]
    end

    FE -- "HTTP" --> EP
    EP --> PM --> DP
    GA --> DP
    DP --> VB
    DP --> TD
    DP --> SAM
    VB --> FUS
    TD --> FUS
    FUS --> DP
    VP --> DP
    DP --> EXP --> GCS --> HU
    HU -- "validated data" --> TRN --> TD

Key Features

• Prompt-driven object detection - detect objects from natural-language descriptions
• Multi-model integration - combine traditional detectors with vision-language models
• Zero-shot capabilities - detect novel objects without prior training
• Hybrid detection modes - parallel fusion or sequential validation of VLM and traditional detections
• SAM segmentation - precise instance masks for every detection
• Video processing - frame extraction, scene-change sampling, and per-frame analysis
• Label Studio integration - export to Label Studio-compatible JSON for human review
• Complete annotation loop - request, detect, annotate, validate, retrain

Quick Start

Prerequisites

• Python 3.8+ (3.11 recommended), CUDA-capable GPU for local VLM inference
• Node.js 18+ for the frontend
• Optional: Ollama for local LLM backends, vLLM for high-performance inference

Installation

git clone https://github.com/rithwik-01/Automate-Annotation.git
cd Automate-Annotation

# Create an environment
conda create --name py312 python=3.12
conda activate py312

# CUDA-enabled PyTorch (adjust to your CUDA version)
conda install cuda -c nvidia/label/cuda-12.6
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

# Install the package (development mode)
pip install -e .

Run the Full Stack

# Terminal 1 - backend (http://localhost:8000, docs at /docs)
cd backend && uvicorn src.main:app --reload --host 0.0.0.0 --port 8000

# Terminal 2 - frontend (http://localhost:5173)
cd frontend && npm install && npm run dev

Or launch the standalone Gradio interface:

python tools/gradio_vlm.py \
  --checkpoint-path Qwen/Qwen2.5-VL-7B-Instruct \
  --backend huggingface \
  --server-port 7860

Usage Examples

Unified VLM Backend

from AutoVisionAnnotatorModels.VLM.vlm_classifierv4 import HuggingFaceVLM
from PIL import Image

vlm = HuggingFaceVLM(
    model_name="Qwen/Qwen2.5-VL-7B-Instruct",
    device="cuda",
)

image = Image.open("path/to/image.jpg")
descriptions = vlm.generate(
    images=[image],
    prompts=["Describe this image in detail"],
)
print(descriptions[0])

Supported architectures include BLIP-2, LLaVA, Qwen2.5-VL (with Flash Attention 2), SmolVLM, and GLM-4.5V/4.1V, each runnable on HuggingFace, OpenAI-compatible APIs, Ollama, or vLLM backends.

Prompt-Driven Object Detection

The Qwen object detection pipeline supports three modes:

from AutoVisionAnnotatorModels.VLM.qwen_object_detection_pipeline3 import QwenObjectDetectionPipeline

pipeline = QwenObjectDetectionPipeline(
    model_name="Qwen/Qwen2.5-VL-7B-Instruct",
    device="cuda",
    enable_sam=True,
    enable_traditional_detectors=True,
    traditional_detectors=["yolo", "detr"],
    vlm_backend="huggingface",
)

results = pipeline.detect_objects_hybrid(
    image_path="path/to/image.jpg",
    use_sam_segmentation=True,
    sequential_mode=False,
    save_results=True,
)

for obj in results["objects"]:
    print(f"{obj['label']}: {obj['bbox']} - {obj['description']}")
print(results["visualization_path"])
print(results["json_path"])

REST API

import requests

BASE = "http://localhost:8000"

filename = requests.post(f"{BASE}/api/upload", files={"file": open("image.jpg", "rb")}).json()["filename"]

description = requests.post(
    f"{BASE}/api/vlm/describe-image/{filename}",
    json={"custom_prompt": "What is in this image?"},
).json()["description"]

results = requests.post(
    f"{BASE}/api/vlm/detect-objects/{filename}",
    json={"detection_method": "Hybrid Mode", "use_sam_segmentation": True},
).json()

Full endpoint reference: backend/VLM_API_REFERENCE.md.

Video Processing

from AutoVisionAnnotatorModels.detectors.videopipeline import VideoPipeline

video_pipeline = VideoPipeline(detector_name="yolov8x")
detections = video_pipeline.process_video(
    video_path="path/to/video.mp4",
    output_path="path/to/output.mp4",
)

Label Studio Export

from AutoVisionAnnotatorModels.export_to_label_studio import export_detections_to_label_studio

export_detections_to_label_studio(
    detections=results,
    image_path="path/to/image.jpg",
    output_path="label_studio_annotations.json",
)

Validated annotations can be stored in Google Cloud Storage (utils/labelstudiogcp.py) and used to fine-tune detectors (multimodels_train.py), closing the annotation loop:

1. Submit a natural-language detection request
2. Pre-process images or video frames
3. Detect automatically with traditional models and/or VLMs
4. Export annotations in Label Studio format
5. Validate and correct annotations with human reviewers
6. Retrain or fine-tune models using the validated data

Evaluation

• Precision, recall, and F1 metrics with per-class breakdowns
• COCO-style evaluation (evaluation.py) and KITTI benchmarks (test_rtdetrv2_kitti.py)
• Side-by-side comparison across detector and VLM configurations
• Visualization of predictions against ground truth

Application: AI City Issue Detection

AutoVision-Annotator powers a real-time urban monitoring system that detects city issues from camera streams:

• Acquisition - integrates city camera feeds, extracts frames via scene-change detection, and stores media in GCS; on-device privacy blurring for faces and license plates
• Detection - YOLO/DETR for common objects plus Grounding DINO + SAM and VLM prompting for novel issues ("find potholes", "detect broken streetlights")
• Classification - categorizes infrastructure damage, traffic violations, safety hazards, and public-space misuse with severity scoring
• Annotation - bounding boxes, masks, captions, and metadata exported for Label Studio validation
• Response - prioritized alerts and issue-tracking dashboards for maintenance teams

Project Structure

Automate-Annotation/
├── backend/                        # FastAPI REST API
│   ├── src/api/                    #   Endpoint routers (VLM, uploads, health, ...)
│   ├── src/pipeline.py             #   Singleton pipeline management
│   └── Dockerfile
├── frontend/                       # React (Vite) user interface
│   └── src/components/             #   Workspace, canvas, model controls, output panel
├── AutoVisionAnnotatorModels/
│   ├── VLM/                        # VLM backends and detection pipelines
│   ├── detectors/                  # YOLO/DETR/RT-DETR, video pipeline, evaluation
│   ├── configs/                    # Class lists and two-step pipeline configs
│   └── utils/                      # GCS, Label Studio, geometry helpers
├── tools/                          # Gradio app, vLLM tests, edge deployment tooling
├── docs/                           # MkDocs documentation and figures
└── static/                         # Uploaded media storage

Documentation

Advanced Setup

pip install vllm

curl -fsSL https://ollama.com/install.sh | sh
ollama pull gpt-oss:20b
ollama run gpt-oss:20b

gcloud init
gcloud auth login
gcloud config set project <project_id>
gcloud auth application-default login

gsutil ls gs://roadsafetytarget/

Troubleshooting

License

This project is released under the MIT License.