Rarámuri

Fast TRIBEv2 inference on NVIDIA Blackwell.

Rarámuri wraps TRIBEv2 in a faster, easier-to-run inference stack for NVIDIA DGX Spark and other Blackwell targets. On GB10, the default path is about 3.3× faster than the FP32 baseline through precision tuning, front-loaded transcription, and model caching, without changing the final predictions in any meaningful way.

What Rarámuri does

Rarámuri predicts voxel-level cortical activation from video. The pipeline runs five models:

1. Parakeet TDT 0.6B (NVIDIA NeMo) — speech-to-text transcription
2. VJEPA2 ViT-G — video feature extraction, 64 frames per clip at 256×256
3. Wav2Vec 2.0 — audio feature extraction
4. Llama 3.2 3B — contextualised word embeddings
5. TRIBEv2 head — maps multimodal features to 20,484-voxel brain predictions

It is heavily based on TRIBEv2, but extensively optimised for Grace Blackwell architectures:

• Mixed precision <3 Blackwell: VJEPA2, Wav2Vec, and Llama run in BF16 unless you override them, which cuts memory traffic and materially improves throughput on GB10 without meaningfully changing the final voxel predictions.
• Optional FP8 quantisation on selected extractors: VJEPA2, Wav2VecBert, and Llama can each be switched to FP8 independently for lower latency when a small amount of quality drift is acceptable.
• Front-loaded CPU transcription: Parakeet transcription starts early on CPU in a background thread, overlapping with GPU model loading and event construction instead of sitting on the critical path.
• Configurable parallel extractor scheduling: text, audio, and optionally video feature extractors can prepare concurrently, while Hugging Face model loads remain serialized behind a lock to avoid safetensor/meta-tensor race conditions.
• Hot-path model persistence: extractor models can stay resident across requests instead of being freed after each stage, which removes repeated reload overhead for server and batch workflows.
• Aggressive warmup and cache seeding: the image preloads spaCy, NLTK, and MNE assets at build time, the server warms TRIBEv2 and runtime dependencies at startup, and extracted features are cached on disk via exca so repeat runs can skip expensive work entirely.

A todo: one day most of this ought to run in NVFP4. Unfortunately, MSLK has other ideas at the moment.

Results

Measured on NVIDIA GB10 (DGX Spark), 130.7 GB unified memory, aarch64.

End-to-end timing (35-second clip, 69 words, 4 sentences)

Per-component breakdown (35-second clip)

* Audio and text extractors run in parallel (mode 1). Wall time for the parallel group is ~50s rather than ~85s sequential.

Previous benchmark (15-second clip, 17 words, 4 sentences)

Precision quality

BF16 predictions were compared against FP32 on the same 35-second input (35 timesteps × 20,484 vertices = 717,940 voxel predictions):

No single voxel prediction differs by more than 0.012 between FP32 and BF16. The activation distributions are virtually identical (mean, std, and range match to 3+ decimal places).

BF16 is the default. FP8 is available as an opt-in when you want lower latency and can tolerate small quality drift.

Supported hardware

Quick start

# Pull the published image
docker pull ghcr.io/chrisvoncsefalvay/raramuri:latest

# Or build it locally
docker build -t ghcr.io/chrisvoncsefalvay/raramuri:latest -f docker/Dockerfile .

# Run inference on a local file (cold start, ~108s for 15s clip)
docker run --gpus all \
  -v /path/to/models:/models \
  -e HF_TOKEN=$HF_TOKEN \
  -e RARAMURI_TRANSCRIPT_BACKEND=parakeet \
  ghcr.io/chrisvoncsefalvay/raramuri:latest \
  /path/to/video.mp4 -o /models/results.json

# Run inference on a YouTube URL (first 30 seconds)
docker run --gpus all \
  -v /path/to/models:/models \
  -e HF_TOKEN=$HF_TOKEN \
  -e RARAMURI_TRANSCRIPT_BACKEND=parakeet \
  ghcr.io/chrisvoncsefalvay/raramuri:latest \
  "https://www.youtube.com/watch?v=..." --start-time 00:00:00 --end-time 00:00:30 \
  -o /models/results.json

# Run the hot server (default port 8765)
docker run --gpus all \
  -v /path/to/models:/models \
  -e HF_TOKEN=$HF_TOKEN \
  -e RARAMURI_TRANSCRIPT_BACKEND=parakeet \
  -e RARAMURI_PERSIST_EXTRACTOR_MODELS=1 \
  -e RARAMURI_VJEPA_DTYPE=bf16 \
  -p 8765:8765 \
  --entrypoint python \
  ghcr.io/chrisvoncsefalvay/raramuri:latest \
  /app/infer_server.py --port 8765

Published image: ghcr.io/chrisvoncsefalvay/raramuri:latest.

The CLI accepts a local file path or any yt-dlp-supported URL as the first positional argument. The server accepts the same sources via the video_path (local) or video_url (remote) fields.

Video ingestion

Remote URLs are downloaded via yt-dlp at up to 720p (MP4). When start_time and/or end_time are provided, yt-dlp's --download-sections is used to download only the requested range — the full video is never fetched. Local files are used directly unless a time range is requested, in which case ffmpeg trims a copy.

The pipeline accepts any source yt-dlp can handle: YouTube, Vimeo, Twitter/X, direct MP4 URLs, and 1,700+ other sites. Time ranges use HH:MM:SS format or bare seconds.

Downloads go to a temporary directory that is cleaned up after inference completes (or fails). Transfer timing metadata is included in the result JSON under timing.transfer.input_prepare.

The image expects a runtime Hugging Face token because meta-llama/Llama-3.2-3B is gated. Startup warmup will fail fast if the token is missing or does not have access.

Server API

The server stays unready (GET /ready returns 503) until all models are warmed.

Both /infer and /jobs accept the same JSON body:

{
  "video_path": "/path/to/local/video.mp4",
  "video_url": "https://www.youtube.com/watch?v=...",
  "start_time": "00:00:00",
  "end_time": "00:00:30",
  "output": "/models/results.json"
}

Provide exactly one of video_path or video_url. start_time, end_time, and output are optional.

Example — synchronous request:

curl http://127.0.0.1:8765/ready
curl http://127.0.0.1:8765/status | jq
curl -X POST http://127.0.0.1:8765/infer \
  -H 'Content-Type: application/json' \
  -d '{"video_path":"/path/to/video.mp4","output":"/models/results.json"}'

Example — async job with progress polling:

# Submit
JOB=$(curl -s -X POST http://127.0.0.1:8765/jobs \
  -H 'Content-Type: application/json' \
  -d '{"video_url":"https://www.youtube.com/watch?v=...","end_time":"00:00:25"}')
JOB_ID=$(echo "$JOB" | jq -r .job_id)

# Poll progress
curl -s "http://127.0.0.1:8765/jobs/$JOB_ID" | jq '{status, phase, step, progress_ratio}'

# Retrieve result when complete
curl -s "http://127.0.0.1:8765/jobs/$JOB_ID/result" | jq .result.shape

Example — composite visualization:

# After a job completes, render a side-by-side video
curl -s -X POST "http://127.0.0.1:8765/jobs/$JOB_ID/visualize" \
  -H 'Content-Type: application/json' \
  -d '{"fps": 24, "width": 1920, "height": 540}' | jq .

# Download the rendered video
FILENAME=$(curl -s -X POST "http://127.0.0.1:8765/jobs/$JOB_ID/visualize" \
  -H 'Content-Type: application/json' -d '{}' | jq -r .download_url)
curl -o composite.mp4 "http://127.0.0.1:8765$FILENAME"

The visualization renders a composite MP4 with the original video (timestamped) on the left, a nilearn glass brain of predicted cortical activations on the right, captions at the bottom left, and a progressively drawn audio spectrum at the bottom right. The 1 Hz brain predictions are linearly interpolated to the output frame rate for smooth playback. Audio is muxed from the original source.

Configuration

Most tuning is exposed through environment variables. The defaults are the recommended GB10 configuration.

Precision knobs

Pipeline knobs

Server knobs

Parallelism modes

• 0 (default): sequential extraction, in order.
• 1: text and audio extract in parallel, then video.
• 2: all extractors run concurrently. Model loading is still serialized behind a lock to avoid Hugging Face safetensor races during concurrent from_pretrained calls.

Architecture

graph TD
    V[Video file] --> FFa[ffmpeg audio extraction]
    V --> FFv[ffmpeg video decode]
    FFa --> P[Parakeet TDT 0.6B<br/>CPU, background thread]
    P --> L[Llama 3.2 3B<br/>GPU · BF16]
    FFa --> W[Wav2VecBert 2.0<br/>GPU · BF16]
    FFv --> VJ[VJEPA2 ViT-G<br/>GPU · BF16<br/>30 clips × 2.0s]
    L --> T[TRIBEv2 head]
    W --> T
    VJ --> T
    T --> R["Predictions<br/>(16 × 20,484 vertices)"]

Parakeet runs on CPU in a background thread and overlaps with GPU model loading. GPU models run in BF16 by default. Feature extraction results are cached to disk via exca, so repeated analysis of the same video can skip extraction entirely.

Monitoring

The /metrics endpoint serves Prometheus-compatible text/plain output. Point a Prometheus scrape job at it:

# prometheus.yml
scrape_configs:
  - job_name: raramuri
    scrape_interval: 15s
    static_configs:
      - targets: ["host.docker.internal:8765"]  # or the container's address

Exposed metrics

A useful Grafana alert: raramuri_server_ready == 0 for more than 60 seconds after startup means warm-up failed. Check GET /status for the warm_error field.

Grafana dashboard

Import grafana/raramuri-dashboard.json into Grafana (Dashboards → Import → Upload JSON). Select your Prometheus datasource when prompted. The dashboard includes:

• Service Health row: ready/warming status, uptime, active/pending counts, completed/failed/rejected totals
• Current Request row: progress gauge, elapsed time, pipeline step, last duration, throughput rate
• Memory & GPU row: process RSS and GPU allocated/reserved over time with peak lines
• Request History row: cumulative counters and per-request duration bars color-coded by latency

Caching

Extracted features are cached at $TRIBE_CACHE (default /models/tribe) by exca. Cache keys are derived from the video path and duration, not from model precision. Clear the cache to force re-extraction:

rm -rf $TRIBE_CACHE/neuralset.extractors.*.release/

Repository structure

raramuri/
├── docker/
│   ├── Dockerfile              # Canonical OCI image (GB10 primary, arm64)
│   ├── infer.py                # Inference script with optimisations
│   ├── infer_server.py         # Persistent hot server with sync and async APIs
│   └── preload_assets.py       # Build-time asset preloading
├── grafana/
│   └── raramuri-dashboard.json # Importable Grafana dashboard
├── tests/
│   ├── test_precision.py       # BF16/FP8 quality verification
│   ├── test_inference.py       # End-to-end smoke tests
│   ├── test_infer_unit.py      # infer.py unit tests
│   ├── test_infer_server_unit.py   # Server validation and concurrency tests
│   ├── test_infer_server_http.py   # HTTP handler integration tests
│   ├── test_promise_handling.py    # Future/promise model and job lifecycle tests
│   └── test_server.py          # Hot server health and response tests
└── README.md

Author

I'm Chris von Csefalvay, an AI researcher specialising in post-training, and the author of Post-Training: A Practical Guide for AI Engineers and Developers (No Starch Press, 2026). I also write Post-Slop, a periodic diatribe about AI, and what it's doing for society. You can also find me on LinkedIn and X.

License

TRIBEv2 is released under CC-BY-NC 4.0. This repository includes optimization patches and deployment tooling for TRIBEv2 and inherits that licence for any TRIBEv2-derived components. The surrounding infrastructure code here, such as precision patches, parallelism, and server code, is MIT-licensed where it does not incorporate TRIBEv2 code.

The pipeline uses models with their own licence terms: Llama 3.2 (Meta), VJEPA2 (CC-BY-NC 4.0), Wav2VecBert 2.0 (Apache 2.0), and Parakeet TDT (CC-BY-4.0).