COGS 185 Final Project

Distilling VLM-Derived Indoor Affordance Scores from Synthetic Scene Images

Taggert Smith | Department of Cognitive Science, UC San Diego

This repository contains two integrated systems:

1. project/ -- A research pipeline that trains lightweight models to predict how suitable indoor scenes are for specific activities (affordances), using knowledge distilled from a vision-language model.
2. Image_Tagger_3.5/ -- A full-stack computer vision application for analyzing architectural interior images, now extended with the affordance prediction pipeline from the research experiments.

---

Repository Layout

FINAL/
├── README.md                  ← you are here
├── project/                   ← research experiments & affordance pipeline
│   ├── configs/               ← affordance definitions, COCO class maps, image manifest
│   ├── data/                  ← Hypersim images, segmentation outputs, VLM annotations
│   ├── src/                   ← all pipeline source code
│   │   ├── segmentation/      ← Mask2Former panoptic segmentation
│   │   ├── features/          ← 310-dim feature extraction from segmentation masks
│   │   ├── vlm/               ← Qwen2-VL annotation pipeline + prompts
│   │   ├── models/            ← LightGBM & CNN training scripts
│   │   └── evaluation/        ← 7-experiment evaluation suite
│   ├── outputs/
│   │   ├── models/            ← trained model checkpoints (.pkl)
│   │   ├── results/           ← experiment CSVs (model comparison, ablation, etc.)
│   │   ├── figures/           ← publication-ready plots
│   │   └── report/            ← LaTeX paper (main.tex → main.pdf)
│   └── scripts/               ← data download utilities
│
├── Image_Tagger_3.5/          ← full-stack image analysis application
│   ├── backend/               ← FastAPI + PostgreSQL backend
│   │   ├── science/           ← computer vision pipeline (20+ analyzers)
│   │   ├── api/               ← REST API endpoints
│   │   ├── models/            ← database models (SQLAlchemy)
│   │   ├── services/          ← VLM, auth, storage services
│   │   └── scripts/           ← seeding & training utilities
│   ├── frontend/              ← React monorepo (4 apps)
│   ├── deploy/                ← Docker Compose + Dockerfiles + nginx
│   ├── docs/                  ← deployment & usage guides
│   └── install.sh             ← one-command setup
│
├── planning docs/             ← project plans & taxonomy spreadsheet
└── ml-hypersim-main/          ← Hypersim dataset utilities (reference)

---

Research Experiments (project/)

What it does

The research pipeline answers: Can a lightweight tabular model predict how suitable an indoor scene is for a given activity, using only object-level features from panoptic segmentation?

Five affordances are studied:

Pipeline stages

1. Image acquisition -- 420 images from the Hypersim synthetic indoor dataset, stratified across 7 room clusters.
2. Panoptic segmentation -- Mask2Former (COCO-133 classes) extracts object masks, centroids, and areas.
3. VLM annotation -- Qwen2-VL-7B scores each image-affordance pair on a 1-7 Likert scale and outputs structured semantic indicators.
4. Feature engineering -- 310 raw features (object presence/counts, pairwise distances, room aggregates) plus 1,248 binary indicator features distilled from VLM annotations.
5. Model training -- LightGBM regressors optimized with Optuna (100 trials, 5-fold CV).

Key results

Model D is statistically significantly better than Model B for most affordances (Wilcoxon p < 0.05).

Experiment output files

All results are in project/outputs/results/:

The paper is at project/outputs/report/main.pdf.

---

Image Tagger 3.5 (Image_Tagger_3.5/)

Image Tagger is a Docker-based web application for analyzing indoor architectural images. It runs a layered computer vision pipeline and provides a browser UI for exploring results.

Prerequisites

• Docker and Docker Compose (v2+)
• At least 8 GB RAM available for Docker (the OneFormer segmentation model is large)
• Optionally, one or more VLM API keys for higher-level analysis:
  • GEMINI_API_KEY (Google Gemini Flash -- recommended, cheapest)
  • OPENAI_API_KEY (GPT-4o)
  • ANTHROPIC_API_KEY (Claude)

Quick Start

cd Image_Tagger_3.5
bash install.sh

This single command will:

1. Check that Docker is installed.
2. Run structural integrity checks (Guardian governance scripts).
3. Build and start three Docker containers (PostgreSQL, FastAPI backend, React + Nginx frontend).
4. Seed the database with the attribute taxonomy and VLM model configurations.
5. Run smoke tests to verify everything is working.

Once complete, open your browser to:

The default entry point (http://localhost:8080/) redirects to the Explorer.

Architecture Overview

Browser (:8080)
  │
  └─ Nginx reverse proxy
       ├─ /explorer/   → React Explorer SPA
       ├─ /workbench/  → React Workbench SPA
       ├─ /admin/      → React Admin SPA
       ├─ /monitor/    → React Monitor SPA
       └─ /api/        → FastAPI backend (:8000)
                             │
                             ├─ Science Pipeline (20+ analyzers)
                             │   ├─ L0: Color, texture, complexity, fractals
                             │   ├─ L1: Depth, spatial frequency, fluency
                             │   ├─ L1.5: OneFormer segmentation
                             │   ├─ L1.8: Affordance prediction (NEW)
                             │   └─ L2: Cognitive/affective VLM analysis
                             │
                             └─ PostgreSQL 15

The Science Pipeline

The backend runs a layered analysis pipeline on each image. Lower layers are fast heuristics; higher layers use deep learning or VLM calls.

Each analyzer writes normalized attributes to the frame, which are then persisted to the database and visible in the Explorer UI.

Enabling Affordance Prediction

The affordance analyzer is opt-in. It requires the OneFormer segmentation layer (L1.5) to run first. The Explorer UI also computes and caches affordance scores on demand for image cards and the image detail view.

From the API / debug UI:

Visit http://localhost:8080/api/v1/debug/images/{image_id}/affordance to get a JSON response with affordance scores for any image in the database. The Explorer also exposes GET /api/v1/explorer/images/{image_id}/affordance for the GUI.

From Python code:

from backend.science.pipeline import SciencePipeline, SciencePipelineConfig

config = SciencePipelineConfig()
config.enable_segmentation = True   # required -- runs OneFormer
config.enable_affordance = True     # enables affordance prediction

pipeline = SciencePipeline(db=session, config=config)
pipeline.process_image(image_id)

# Scores are saved to the Validation table as:
#   affordance.L059       (1.0-7.0 Likert scale)
#   affordance.L059_norm  (0.0-1.0 normalized)
#   ... etc for L079, L091, L130, L141

Standalone (no database):

from backend.science.context.affordance import predict_affordances_from_image
import cv2

image = cv2.cvtColor(cv2.imread("my_room.jpg"), cv2.COLOR_BGR2RGB)
scores = predict_affordances_from_image(image)
# {'L059': 4.73, 'L079': 0.95, 'L091': 2.51, 'L130': 3.11, 'L141': 2.36}

Retraining the Affordance Models

The pre-trained LightGBM models are bundled in backend/science/data/affordance_models/. To retrain from the pilot dataset (e.g., after updating training data):

cd Image_Tagger_3.5
pip install lightgbm pandas optuna   # if not already installed
python -m backend.scripts.train_affordance_models

This reads project/data/assembled_dataset/pilot_dataset.parquet, trains one LightGBM regressor per affordance with Optuna hyperparameter optimization (50 trials, 5-fold CV), and saves the models.

User Roles & Authentication

The app uses a simple header-based role system suitable for classroom use:

The default API_SECRET is dev_secret_key_change_me. For any non-local deployment, set a real secret via the SECRET_KEY environment variable in deploy/docker-compose.yml.

VLM Configuration

L2 analyzers (cognitive, semantic, architectural) require a VLM backend. The app auto-detects available providers from environment variables. Set one or more in deploy/docker-compose.yml:

environment:
  - GEMINI_API_KEY=your_key_here      # cheapest option
  - OPENAI_API_KEY=your_key_here      # GPT-4o
  - ANTHROPIC_API_KEY=your_key_here   # Claude

A built-in cost tracker enforces a hard budget limit (default $15 USD, configurable via VLM_HARD_LIMIT_USD) to prevent runaway API spending.

The affordance analyzer (L1.8) now prefers the best-performing project model: indicator-augmented LightGBM (Model D). That path uses OneFormer segmentation plus runtime VLM indicator extraction when a VLM API key is configured. If no VLM is configured, the app falls back to the bundled raw-feature LightGBM model.

Stopping & Restarting

# Stop all containers (data is preserved in Docker volumes)
cd Image_Tagger_3.5/deploy
docker compose down

# Restart
docker compose up -d

# Full reset (deletes database and image data)
docker compose down -v

Development (without Docker)

For local development of the backend:

cd Image_Tagger_3.5

# Start PostgreSQL separately (or use a local instance)
# Set DATABASE_URL in your environment

# Install Python dependencies
pip install fastapi uvicorn sqlalchemy psycopg2-binary torch transformers \
            lightgbm pandas scikit-image scipy supervision

# Run the backend
uvicorn backend.main:app --reload --port 8000

# In another terminal, run the frontend dev servers
cd frontend
npm install
npm run dev:all    # starts all 4 apps on ports 3001-3004

Useful API Endpoints

---

Other Directories