plm-captioner

plm-captioner is a FastAPI microservice that runs a configurable PLM-3B-style language model to produce Action100M-compatible captions and action labels for temporal video segments.

• Input: Action100M-style segments (node_id, parent_id, level, start, end) plus optional llama3_caption and video-level metadata.
• Model: Any Hugging Face causal LM (default: Qwen/Qwen2.5-3B-Instruct) running on GPU (CUDA) with configurable dtype and optional 4-bit quantization.
• Output: For each segment, this service fills:
  • plm_caption – free-form natural language caption.
  • plm_action – short action label / verb phrase (e.g. "open journal", "stir coffee").

This service is designed to plug into the broader Mind and Motion Labs pipeline alongside:

• vjepa-segmenter → temporal segments
• plm-captioner → PLM captions and action labels (plm_caption, plm_action)
• llama-captioner → mid-frame captions (llama3_caption)
• gptoss-reasoner → structured reasoning annotations (gpt)
• maml-api → orchestrator and public API

Prompt note (Action100M parity)

The segment-level PLM prompt starts with the exact paper phrasing:

• "Describe this video in detail."

The service then asks for structured JSON so plm_caption and plm_action can be threaded through the API pipeline reliably.

---

1. API Overview

• GET /health – health + configuration of this service.
• POST /plm – run PLM captioning + action labeling on a batch of segments.

GET /health

Example response:

{
  "status": "ok",
  "model_name": "Qwen/Qwen2.5-3B-Instruct",
  "device": "cuda:0",
  "dtype": "fp16",
  "quantization": null,
  "timestamp": "2024-01-01T12:00:00Z"
}

POST /plm

Request body (JSON, minimal example):

{
  "segments": [
    {
      "node_id": "0",
      "parent_id": null,
      "level": 0,
      "start": 0.0,
      "end": 10.0,
      "llama3_caption": "optional existing caption",
      "gpt": null
    }
  ],
  "metadata": {
    "video_id": "optional-id",
    "source": "egocentric-100k",
    "language": "en"
  }
}

• Extra fields on each segment are accepted and preserved (extra="allow").
• llama3_caption is optional and, when present, is used as additional context.

Response body (JSON):

{
  "request_id": "a4b1e3d4-1234-5678-9abc-def012345678",
  "status": "completed",
  "segments": [
    {
      "node_id": "0",
      "parent_id": null,
      "level": 0,
      "start": 0.0,
      "end": 10.0,
      "plm_caption": "A person opens their journal on a white desk and starts to arrange paper pieces.",
      "plm_action": "open journal",
      "llama3_caption": "optional existing caption",
      "gpt": null
    }
  ]
}

The service preserves all existing keys per segment and only adds/updates plm_caption and plm_action.

---

2. Configuration

Configuration is managed via environment variables (prefix PLM_) and app/config.py:

• Model:
  • PLM_MODEL_NAME – Hugging Face model name (default: Qwen/Qwen2.5-3B-Instruct).
  • PLM_DEVICE – "cuda" (default) or "cpu".
  • PLM_DTYPE – "fp16" (default), "bf16", or "fp32".
  • PLM_QUANTIZATION – "4bit" to enable BitsAndBytes 4-bit loading (optional).
• Generation:
  • PLM_MAX_NEW_TOKENS – max new tokens per segment prompt (default: 128).
  • PLM_TEMPERATURE – sampling temperature (default: 0.7).
  • PLM_TOP_P – nucleus sampling top_p (default: 0.9).
• Runtime:
  • PLM_REQUEST_TIMEOUT_SECONDS – logical timeout budget for requests (unused by default, reserved).
  • PLM_MAX_BATCH_SIZE – max segments processed per HF generation batch (default: 32).
  • PLM_LOG_LEVEL – logging level (INFO, DEBUG, etc.; default: INFO).

All environment variables are optional; sane defaults are provided for local development.

---

3. Local Development (uv, venv only)

All Python dependencies are installed into a local virtual environment using uv. The host Python installation is not modified.

See docs/SETUP.md for full step-by-step instructions. In short:

git clone https://github.com/ankk98/plm-captioner.git
cd plm-captioner

# Create .venv with Python 3.12
uv venv --python 3.12
source .venv/bin/activate   # Linux/macOS

# Install project
uv pip install -e .

# (Optional) set configuration
export PLM_MODEL_NAME="Qwen/Qwen2.5-3B-Instruct"
export PLM_DEVICE="cuda"
export PLM_DTYPE="fp16"

# Run FastAPI app
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

Test:

curl http://localhost:8000/health

curl -X POST http://localhost:8000/plm \
  -H "Content-Type: application/json" \
  -d '{
    "segments": [
      {
        "node_id": "0",
        "parent_id": null,
        "level": 0,
        "start": 0.0,
        "end": 10.0,
        "llama3_caption": "Hands open a black journal on a white desk.",
        "gpt": null
      }
    ],
    "metadata": {
      "video_id": "demo-001",
      "source": "egocentric-100k",
      "language": "en"
    }
  }'

---

4. Docker (GPU)

The provided Dockerfile builds a container that:

• Uses nvidia/cuda:12.1.0-runtime-ubuntu22.04 as the base image.
• Installs Python 3.12 and uv inside the container.
• Creates a virtual environment and installs plm-captioner there (no host pollution).
• Exposes FastAPI on port 8000.

Build and run:

docker build -t plm-captioner:latest .

docker run --rm -p 8000:8000 \
  --gpus all \
  -e PLM_MODEL_NAME="Qwen/Qwen2.5-3B-Instruct" \
  -e PLM_DEVICE="cuda" \
  plm-captioner:latest

Example docker-compose service (for reference, matching maml-api style):

plm-captioner:
  image: ankk98/plm-captioner:latest
  container_name: plm
  environment:
    - PLM_MODEL_NAME=Qwen/Qwen2.5-3B-Instruct
    - PLM_DEVICE=cuda
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities: ["gpu"]
  healthcheck:
    test: ["CMD-SHELL", "curl -fsS http://localhost:8000/health || exit 1"]
    interval: 30s
    timeout: 10s
    retries: 3
    start_period: 30s

---

5. Integration with maml-api

The service is designed to be called as a PLM-3B stage between V-JEPA segmentation and downstream captioning / reasoning:

1. vjepa-segmenter produces temporal segments (nested tree, flattened to node_id, parent_id, level, start, end by maml-api).
2. plm-captioner is called with those segments to fill plm_caption and plm_action.
3. llama-captioner attaches llama3_caption mid-frame captions.
4. gptoss-reasoner takes plm_* + llama3_caption and produces gpt.summary and gpt.action.
5. maml-api maps the final segments into Action100MNode objects and returns an AnnotationResult.

maml-api already expects plm_caption and plm_action fields on each segment when mapping to Action100MNode, so as long as its PLM client calls this service and threads the updated segments through the pipeline, the final API will produce Action100M-style annotations with PLM fields filled.

---

6. Component validation (standalone)

Validate this service independently before full pipeline tests:

# Health
curl -s http://localhost:8000/health | jq .

# Functional
curl -s -X POST http://localhost:8000/plm \
  -H "Content-Type: application/json" \
  -d '{
    "segments":[{"node_id":"0","parent_id":null,"level":0,"start":0.0,"end":10.0}],
    "metadata":{"source":"manual-test","language":"en"}
  }' | jq .

Expected:

• Response status is completed
• segments[0].plm_caption exists
• segments[0].plm_action exists