VLM-EPI Pipeline

Modular Vision-Language Model pipeline for Personal Protective Equipment (PPE) detection in industrial environments.

What it does

Runs Google's Gemma 4 vision-language model on edge devices (Raspberry Pi, smartphones, Jetson) to analyze worker safety in industrial environments — detecting whether construction or plant workers are wearing required Personal Protective Equipment (PPE) like helmets, vests, gloves, and safety harnesses.

Three output modes:

• Free-form analysis — natural language description of PPE usage per worker
• Structured JSON — schema-validated classification ready to integrate with monitoring systems
• Bounding box detection — grounded localization of PPE items in the image

Built with anti-hallucination safeguards (structural + semantic validation) because false negatives in safety systems have real human cost.

Why it matters

PPE compliance monitoring at hydroelectric plants, construction sites, and industrial facilities is traditionally done by human supervisors or specialized object detectors (YOLO, etc.). VLMs offer a complementary path: contextual reasoning, multi-class understanding, and natural-language alerts — but they need careful engineering to be reliable in safety-critical contexts.

This pipeline explores that engineering: how to wrap a general-purpose VLM into a production-grade, hallucination-resistant system suitable for edge deployment.

Quick start

Dev machine and Jetson Orin Nano (HF Transformers backend)

# Install with HuggingFace Transformers backend (requires CUDA or CPU torch)
pip install -e "./pipelines/vlm_epi[hf,dev]"

# Free-form PPE description
vlm-epi-single worker.jpg --prompt P1_livre --backend hf-transformers

# Structured JSON output (default model: google/gemma-4-E2B-it)
vlm-epi-single worker.jpg --prompt P2_json --backend hf-transformers

# Bounding box detection, custom model
vlm-epi-single worker.jpg --prompt P3_bbox --backend hf-transformers \
    --model-id google/gemma-4-E2B-it

# Separate result JSON from latency log
vlm-epi-single worker.jpg --prompt P2_json --backend hf-transformers \
    > result.json

Raspberry Pi 5 (LiteRT-LM backend)

The LiteRT-LM backend uses Google's edge inference engine and runs on CPU — no GPU required. The ai-edge-litert-lm package and a pre-converted .litertlm model file are needed.

Status: end-to-end validated locally on Linux x86_64 (i5 10th gen, CPU-only) with litert-lm backend as of 2026-05-13 — 57 s total latency for 1 image, valid JSON output.

# Install with LiteRT-LM backend (CPU-only, no torch)
pip install -e "./pipelines/vlm_epi[litertlm,dev]"

# Download the CPU generic variant (~1.5 GB)
hf download litert-community/gemma-4-E2B-it-litert-lm \
    --local-dir ~/models/gemma4-e2b-litert \
    --include "*.litertlm"

# Run — --model-id is required and must point to the .litertlm file
vlm-epi-single worker.jpg --prompt P2_json --backend litert-lm \
    --model-id ~/models/gemma4-e2b-litert/gemma-4-E2B-it.litertlm

# Verbose mode (prints backend info and parse details to stderr)
vlm-epi-single worker.jpg --prompt P3_bbox --backend litert-lm \
    --model-id ~/models/gemma4-e2b-litert/gemma-4-E2B-it.litertlm \
    --verbose

See docs/RPI_SETUP.md for the full Raspberry Pi 5 setup guide, including OS flashing, Python version management, and benchmark replication steps.

Coming in next iterations

• Batch processing (vlm-epi-batch): run all three prompts over a directory of images and aggregate results to a JSON file.
• Full validation (Level 1 + 2): structural schema checks and semantic anti-hallucination guards to replace the current minimal status-propagation pass.

Architecture

flowchart LR
    A[Capture] --> B[Preprocess]
    B --> C[Inference]
    C --> D[Parsing]
    D --> E[Validation]
    E --> F[Output]

    A -.-> A1[file or directory]
    C -.-> C1[HF Transformers,<br/>Ollama,<br/>LiteRT-LM]
    E -.-> E1[Level 1: structural<br/>Level 2: semantic]

Each stage is a pluggable module with a typed dataclass contract. Backends can be swapped without touching the rest of the pipeline.

Project structure

.
├── pipelines/vlm_epi/        # Core pipeline package
│   ├── vlm_epi/              # Source modules
│   ├── scripts/              # CLI entry points
│   └── tests/                # Unit tests with fixtures
├── docs/                     # Architecture decisions, RPi setup guide, etc.
└── pyproject.toml            # Package metadata and dependencies

Tech stack

• Python 3.11+ with full type hints
• Hugging Face Transformers for the reference backend
• Pillow + NumPy for image processing
• pytest for testing, ruff for linting, mypy for type checking
• GitHub Actions for CI

Roadmap

• Project structure, schemas, prompt templates, pyproject
• Parsing module (P1/P2/P3 parsers)
• Inference module with HF Transformers backend and LiteRT-LM backend
• Capture + preprocessing modules
• Pipeline orchestrator + CLI (vlm-epi-single)
• Test suite with fixtures (57 tests)
• Validation module (Level 1 + 2 anti-hallucination)
• Batch CLI (vlm-epi-batch) + directory processing
• Raspberry Pi 5 deployment + benchmark replication
• Ollama backend (alternate local engine)
• Quantitative evaluation against full validation dataset

Context

This work is part of the SafeAI initiative — a partnership between CEIA (Center of Excellence in Artificial Intelligence) and a hydroelectric plant in Foz do Iguaçu, Brazil — for PPE detection in industrial environments. This repository covers the mobile-cameras subteam's exploration of VLMs as a complement to the existing YOLO-based detection pipeline.

License

Apache 2.0 — see LICENSE for details.

Same license as Gemma 4, by design.