🛡️ CompAgent: An Agentic Framework for Visual Compliance Verification

Rahul Ghosh, Baishali Chaudhury, Hari Prasanna Das, Meghana Ashok, Ryan Razkenari, Long Chen, Sungmin Hong, Chun-Hao Liu

Accepted at CVPR 2026 GRAIL-V (Grounded Retrieval and Agentic Intelligence for Vision-Language) Workshop

⚠️ Notice: This code is being released solely for academic and scientific reproducibility purposes, in support of the methods and findings described in the associated publication. Pull requests are not being accepted in order to maintain the code exactly as it was used in the paper.

---

📖 Introduction

CompAgent is the first agentic framework for visual compliance verification — a critical yet underexplored problem in computer vision, especially in domains such as media, entertainment, and advertising where content must adhere to complex and evolving policy rules.

Existing methods often rely on task-specific deep learning models trained on manually labeled datasets, which are costly to build and limited in generalizability. While recent Multimodal Large Language Models (MLLMs) offer broad real-world knowledge and policy understanding, they struggle to reason over fine-grained visual details and apply structured compliance rules effectively on their own.

CompAgent addresses this by augmenting MLLMs with a suite of specialized visual tools — including Amazon Rekognition, Bedrock Data Automation (BDA), LlavaGuard, SafeCLIP, and ICM-Assistant — and introduces a planning agent built on LangGraph that dynamically selects appropriate tools based on the compliance policy. A compliance verification agent then integrates image content, tool outputs, and policy context to perform multimodal reasoning using a ReAct (Reasoning + Acting) paradigm.

Key Results:

• 🏆 Achieves up to 76% F1 score on the UnsafeBench dataset
• 📈 10% improvement over the state-of-the-art
• ✅ Outperforms specialized classifiers, direct MLLM prompting, and curated routing baselines

---

🏗️ Architecture

Figure: Overview of the CompAgent framework. A planning agent dynamically selects visual tools based on the compliance policy, and a compliance verification agent integrates multimodal evidence to produce structured safety assessments.

---

🎬 Demo

Demo: CompAgent in action — dynamically selecting tools and performing visual compliance verification.

---

🖥️ Streamlit Demo App

CompAgent includes an interactive Streamlit UI that provides a lightweight interface to run the LangGraph ReAct agent for image compliance checking with real-time execution trace visualization.

Features

• Model Selection — Choose from Claude 3.5 Sonnet, Claude 3.7 Sonnet, Claude Sonnet 4, Llama 3.3, and more
• Real-time Traces — Watch the agent's reasoning and tool calls as they execute
• Customizable Policies — Edit safety policies directly in the UI
• Final Assessment Display — View structured compliance results (rating, category, rationale)

Running the Streamlit App

1. Ensure metadata is extracted (see Step 1 below)

2. Navigate to the agent directory:

   cd src/agent
   

3. Run the Streamlit app:

   streamlit run streamlit_app.py
   

4. Open your browser at http://localhost:8501

Configuration

The app automatically detects the dataset configuration from BUCKET_NAME in compliance_tools_standalone_langgraph.py:

• For LlavaGuard benchmark: Uses llavaguard_agent.txt prompt
• For UnsafeBench benchmark: Uses unsafebench_agent.txt prompt

---

📁 Repository Structure

grip-compagent/
├── README.md                          # This file
├── requirements.txt                   # Python dependencies (pip)
├── pyproject.toml                     # Project configuration (uv)
├── assets/
│   ├── main_architecture.png          # Architecture diagram
│   ├── demo.gif                       # Demo GIF (for README)
│   └── demo_video.mp4                 # Full demo video
├── src/
│   ├── agent/                         # 🧠 Main CompAgent Framework
│   │   ├── langGraph_reAct_llavaguard.ipynb   # LangGraph ReAct agent for LlavaGuard benchmark
│   │   ├── langGraph_reAct_unsafebench.ipynb  # LangGraph ReAct agent for UnsafeBench benchmark
│   │   ├── compliance_tools_standalone_langgraph.py  # Tool definitions for the agent
│   │   ├── streamlit_app.py           # Interactive Streamlit demo UI
│   │   ├── run_notebook.sh            # Shell script to execute the notebook
│   │   └── prompt/                    # Agent system prompts
│   │       ├── llavaguard_agent.txt   # LlavaGuard policy prompt for agent
│   │       ├── llavaguard.txt         # LlavaGuard standalone prompt
│   │       ├── unsafebench_agent.txt  # UnsafeBench policy prompt for agent
│   │       └── unsafebench.txt        # UnsafeBench standalone prompt
│   ├── tool_run/                      # 🔧 Standalone Tool Runners
│   │   ├── tool_llavaguard.py         # Run LlavaGuard inference on images
│   │   ├── tool_BDA.py               # Run Bedrock Data Automation on images
│   │   ├── tool_rekognition.py        # Run Amazon Rekognition on images
│   │   ├── tool_safeclip.py           # Run SafeCLIP on images
│   │   ├── tool_icm.py               # Run ICM-Assistant on images
│   │   ├── config_icm.yaml           # ICM-Assistant model/runtime config
│   │   └── prompts_icm.yaml          # ICM-Assistant prompt templates
│   ├── baseline_run/                  # 📊 Baseline Comparisons
│   │   ├── run_llama_baseline.ipynb   # Llama baseline
│   │   ├── run_mistral_baseline.ipynb # Mistral/Pixtral baseline
│   │   ├── run_sonnet_baseline.ipynb  # Claude Sonnet baseline
│   │   └── run_notebooks.sh          # Run all baselines
│   ├── evaluation/                    # 📈 Evaluation Notebooks
│   │   ├── evaluation.ipynb           # General evaluation
│   │   ├── evaluation-llavaguard.ipynb# LlavaGuard benchmark evaluation
│   │   └── evaluation-unsafebench.ipynb # UnsafeBench benchmark evaluation
│   ├── prompts/                       # 📝 System Prompts
│   │   ├── prompt_agent_system_w_verification.txt
│   │   ├── prompt_cluster_routing.txt
│   │   └── prompt_llavaguard.txt
│   ├── routing/                       # 🔀 Routing Logic
│   │   ├── compliance_state.py        # Compliance state definitions
│   │   ├── compliance_tools.py        # Tool routing implementations
│   │   └── routing_decisions.txt      # Routing decision documentation
│   └── utils/                         # 🛠️ Utilities
│       ├── bedrock_invoker.py         # AWS Bedrock API wrapper
│       ├── helpers.py                 # General helper functions
│       ├── inference_helpers.py       # Inference utility functions
│       └── openai_clip.py            # OpenAI CLIP utilities

---

⚙️ Environment Setup

Prerequisites

• Python 3.10+
• AWS Account with access to Amazon Bedrock, Amazon Rekognition, and S3
• GPU Instance (required for LlavaGuard, ICM-Assistant, and SafeCLIP tools)
  • 💡 Recommended: We recommend using an AWS g5.xlarge instance (NVIDIA A10G GPU, 24 GB VRAM) to run the GPU-dependent tools.

Installation (Option A: pip)

1. Clone the repository:

   git clone https://gitlab.aws.dev/genaiic-reusable-assets/other/grip-compagent
   cd grip-compagent
   

2. Create and activate a virtual environment:

   python -m venv venv
   source venv/bin/activate
   

3. Install dependencies:

   pip install -r requirements.txt
   

4. Install ICM-Assistant (LLaVA):

   ICM-Assistant depends on the LLaVA package. Create the icm-assistant folder inside src/tool_run/, clone LLaVA into it, and install:

   mkdir -p src/tool_run/icm-assistant
   cd src/tool_run/icm-assistant
   git clone https://github.com/haotian-liu/LLaVA.git
   cd LLaVA
   pip install -e .
   cd ../../../..
   

   The ICM model used is liuhaotian/llava-v1.6-mistral-7b (or as configured in src/tool_run/config_icm.yaml). It will be downloaded automatically by the transformers library on first run.

Installation (Option B: uv)

Alternatively, you can use uv for faster dependency management:

1. Install uv:

   curl -LsSf https://astral.sh/uv/install.sh | sh
   

2. Install dependencies and sync environment:

   uv sync
   

3. Activate the virtual environment:

   source .venv/bin/activate
   

4. Install ICM-Assistant (LLaVA):

   ICM-Assistant depends on the LLaVA package. Create the icm-assistant folder inside src/tool_run/, clone LLaVA into it, and install:

   mkdir -p src/tool_run/icm-assistant
   cd src/tool_run/icm-assistant
   git clone https://github.com/haotian-liu/LLaVA.git
   cd LLaVA
   pip install -e .
   cd ../../../..
   

   The ICM model used is liuhaotian/llava-v1.6-mistral-7b (or as configured in src/tool_run/config_icm.yaml). It will be downloaded automatically by the transformers library on first run.

Configure AWS Credentials

Configure AWS credentials:

export AWS_PROFILE=your-profile
# or
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export AWS_DEFAULT_REGION=us-east-1

---

📦 Dataset Setup

CompAgent is evaluated on two public safety benchmarks. You will need to obtain the datasets and upload them to an S3 bucket.

Supported Benchmarks

S3 Bucket Structure

Create an S3 bucket and upload your images:

# Create bucket
aws s3 mb s3://your-bucket-name --region us-east-1

# Upload images for LlavaGuard benchmark
aws s3 cp ./your-images/ s3://your-bucket-name/testset/ --recursive

# Or for UnsafeBench benchmark
aws s3 cp ./your-images/ s3://your-bucket-name/test/images/ --recursive

Update the BUCKET_NAME variable in src/agent/compliance_tools_standalone_langgraph.py to match your bucket name.

Note: The SafeCLIP tool (src/utils/openai_clip.py) will download the HQ-CLIP model (~1.7 GB) from HuggingFace on first run.

---

🔧 Step 1: Metadata Extraction

Before running the CompAgent framework, you need to extract metadata from images using the standalone tools in src/tool_run/. Each tool processes images from an S3 bucket and uploads the resulting metadata back to S3 directly.

Run LlavaGuard (GPU Required)

python src/tool_run/tool_llavaguard.py \
  --s3_image_path s3://your-bucket/testset/ \
  --s3_output s3://your-bucket/llavaguard_metadata/

Run Bedrock Data Automation (BDA)

python src/tool_run/tool_BDA.py \
  --bucket_name your-bucket-name \
  --prefix_input testset \
  --prefix_output BDA_metadata

Run Amazon Rekognition

python src/tool_run/tool_rekognition.py \
  --bucket_name your-bucket-name \
  --prefix_input testset \
  --prefix_output rekognition_metadata

Run ICM-Assistant (GPU Required)

ICM-Assistant uses the ICM-LLaVA model for image content moderation with configurable prompts via YAML.

Basic S3 Usage:

python src/tool_run/tool_icm.py \
  --s3_image_path s3://your-bucket/testset/ \
  --s3_output s3://your-bucket/icm_assistant_metadata/

Local folder processing:

python src/tool_run/tool_icm.py \
  --local_image_folder ./input_images \
  --local_output ./icm_metadata

Using different prompts (configured in prompts_icm.yaml):

# Use UnsafeBench 12-category prompt
python src/tool_run/tool_icm.py \
  --s3_image_path s3://your-bucket/testset/ \
  --s3_output s3://your-bucket/icm_metadata/ \
  --prompt_name unsafe_bench

# Available: llava_default, unsafe_bench, simple, violence_only, nsfw

Note:

• ICM-Assistant requires the LLaVA package. Create the icm-assistant folder and install: mkdir -p src/tool_run/icm-assistant && cd src/tool_run/icm-assistant && git clone https://github.com/haotian-liu/LLaVA.git && cd LLaVA && pip install -e .
• Configuration files: src/tool_run/config_icm.yaml and src/tool_run/prompts_icm.yaml

Run SafeCLIP (GPU Required)

python src/tool_run/tool_safeclip.py \
  --s3_image_path s3://your-bucket/testset/ \
  --s3_output s3://your-bucket/safeclip_metadata/

Refer to the SafeCLIP integration in src/utils/openai_clip.py for zero-shot toxic image detection.

After running all tools, the extracted metadata (JSON files) will be available in S3 under their respective prefixes:

s3://your-bucket-name/
├── testset/                              # Input images
├── tools/
│   ├── rekognition_metadata/             # Rekognition output (faces, labels, moderation, text)
│   ├── BDA_metadata/                     # BDA output (summary, IAB categories, moderation)
│   ├── llavaguard_metadata/              # LlavaGuard output (rating, category, rationale)
│   ├── icm_assistant_metadata/           # ICM-Assistant output
│   ├── safeclip_prompting_results/       # SafeCLIP output (safe/unsafe scores)
│   ├── llavaguard_validation/            # (Optional) LlavaGuard validation confidence scores
│   ├── safeclip_validation/              # (Optional) SafeCLIP validation confidence scores
│   └── icm_assistant_validation/         # (Optional) ICM validation confidence scores
└── merged_test_with_s3.csv               # Ground truth CSV (id, s3_location, policy, label)

Note: The *_validation/ prefixes are optional. If not present, the agent tools will still function but without confidence validation scores.

---

🚀 Step 2: Run CompAgent Framework

Once metadata is extracted and uploaded to S3, you can run the main CompAgent framework using the LangGraph ReAct agent.

Option A: Run via Jupyter Notebook

For the LlavaGuard benchmark (9-category classification):

cd src/agent
jupyter notebook langGraph_reAct_llavaguard.ipynb

For the UnsafeBench benchmark (binary safe/unsafe classification):

cd src/agent
jupyter notebook langGraph_reAct_unsafebench.ipynb

Option B: Run via Shell Script

cd src/agent
bash run_notebook.sh

This will:

1. Convert the notebook to a Python script
2. Execute the CompAgent ReAct agent
3. Process images through the planning agent and compliance verification agent
4. Output structured safety assessments (rating, category, rationale)

---

📊 Step 3: Evaluation

After running the CompAgent framework, use the evaluation notebooks to compute metrics, generate classification reports, and analyze policy responsiveness.

Comprehensive Evaluation with Policy Analysis

cd src/evaluation
jupyter notebook evaluate.ipynb

This is the primary evaluation notebook that provides comprehensive analysis including:

Overall Evaluation:

• Loads ground truth from CSV and predictions from agent output JSON files
• Computes accuracy, confusion matrices, and classification reports
• Compares CompAgent vs. LlavaGuard baseline performance

Category-wise Evaluation:

• Breaks down performance by safety category (e.g., violence, sexual content, hate speech)
• Identifies which categories the agent performs best/worst on

Policy Responsiveness Analysis:

• Policy Exception Rate (PER): Measures accuracy on images where the safety label changes based on policy variations
• Balanced Accuracy: Accounts for class imbalance in the dataset
• Policy Exception Score (PES): Harmonic mean of PER and Balanced Accuracy
• Accuracy breakdown by policy version (original, v2, v3_a, v3_b, v3_c)
• Generates radar charts for policy responsiveness visualization

Evaluate on LlavaGuard Benchmark

cd src/evaluation
jupyter notebook evaluation-llavaguard.ipynb

This notebook evaluates 9-category safety classification:

• Loads predictions from the agent output JSON files
• Compares against ground truth labels
• Generates confusion matrices and classification reports

Evaluate on UnsafeBench Benchmark

cd src/evaluation
jupyter notebook evaluation-unsafebench.ipynb

This notebook evaluates binary safe/unsafe classification:

• Parses agent predictions (handles JSON, XML, and refusal patterns)
• Computes Precision, Recall, F1 Score, and Accuracy
• Visualizes results with confusion matrix heatmaps

Metrics Computed

Note: The evaluation notebooks require numpy, scikit-learn, seaborn, and matplotlib, which are included in requirements.txt.

---

🙏 Acknowledgements

The authors would like to thank their colleague Isaac Privitera for his thorough review and insightful feedback, which significantly improved the quality and clarity of this research. They also acknowledge the leadership team of Vidya Sagar Ravipati, Wan Chen, Anila Joshi, Atanu Roy, Sri Elaprolu, and Taimur Rashid for their guidance and support on this research.

---

📄 Citation

If you find this work useful, please cite our paper:

📎 Paper: https://arxiv.org/abs/2511.00171

@inproceedings{ghosh2026compagent,
  title={CompAgent: An Agentic Framework for Visual Compliance Verification},
  author={Ghosh, Rahul and Chaudhury, Baishali and Das, Hari Prasanna and Ashok, Meghana and Razkenari, Ryan and Chen, Long and Hong, Sungmin and Liu, Chun-Hao},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  note={GRAIL-V Workshop},
  year={2026}
}

---

⭐ If you find this project helpful, please consider giving it a star!