riprap-inference

Model-serving backend for Riprap. Hosts the specialist models a flood-exposure briefing calls into — entity extraction, embeddings, time-series forecasting, and two geospatial foundation models — behind a small, consistent HTTP contract.

Each model is a LitServe LitAPI: plain PyTorch/HuggingFace code with a setup()/predict() lifecycle, served over FastAPI. No Triton, no gRPC, no .pbtxt config, no CUDA-only server binary. Runs on CUDA, Apple Silicon (MPS), or CPU — wherever LitServe and PyTorch run.

The contract

Riprap's Pebble framework calls into this server through a generic adapter (model_call, in the main repo) that knows nothing about any specific model. It POSTs a JSON body to a path and reads back a JSON response. That's the entire seam:

Riprap  ──POST {path}, JSON body──▶  riprap-inference
Riprap  ◀──JSON response──────────  riprap-inference

Every model returns the same envelope:

{"ok": true, "elapsed_s": 0.4, "device": "mps:0", "...": "model-specific fields"}
{"ok": false, "err": "description of what went wrong"}

ok: false is treated as a pebble going offline, the same as a down upstream API or a missing raster tile — no special-casing needed on the Riprap side. This consistency is what lets the Pebble adapter stay generic instead of growing one code path per model.

Request/response shapes per model are in each LitAPI's docstring in server.py.

Quick start

git clone https://github.com/msradam/riprap-inference
cd riprap-inference
uv venv --python 3.12 && source .venv/bin/activate
uv pip install -r requirements.txt
python server.py                      # binds 0.0.0.0:8000, accelerator=auto
python server.py --port 8001 --accelerator mps

First run downloads model weights from Hugging Face (a few GB). /healthz returns "not ready" until every model has finished loading, then "ok".

Point Riprap at it:

export RIPRAP_ML_BACKEND=remote
export RIPRAP_ML_BASE_URL=http://localhost:8000

Bring your own model

Fork this repo. Add a LitAPI:

class MyModelAPI(ls.LitAPI):
    def setup(self, device):
        self.model = load_my_model().to(device)

    def predict(self, request):
        # request is the raw JSON body Riprap POSTed
        result = self.model(request["input"])
        return {"ok": True, "elapsed_s": ..., "device": self.device, "output": result}

Add it to the LitServer list at the bottom of server.py with an api_path, e.g. /v1/my-model. That's it — no registration step, no schema to update elsewhere. GET /docs picks it up automatically because LitServe is FastAPI underneath.

On the Riprap side, wire it in with a pebble manifest — no code:

adapter: model_call
config:
  path: /v1/my-model
  body:
    input: {source: lat}

See riprap/core/pebbles/adapters/model_call.py in the main repo for the full manifest contract ({source:}/{const:} body templating, response_path extraction, auth headers).

Deployment

The same server.py, unmodified, runs on either target below — this is one codebase, not two implementations to keep in sync.

Modal (scale-to-zero cloud GPU, recommended for a hosted deployment). modal_app.py wraps server.py as a Modal Function via @modal.web_server — LitServe owns its own process model internally (server.run() launches worker processes and a request queue), so this runs the whole process as a black-box HTTP server rather than trying to extract an ASGI app.

modal secret create riprap-inference-secret \
    RIPRAP_INFERENCE_API_KEY=$(openssl rand -hex 24) --env riprap
modal deploy modal_app.py --env riprap

Point Riprap at the printed *.modal.run URL with RIPRAP_ML_BACKEND=remote, RIPRAP_ML_BASE_URL=<url>, RIPRAP_ML_API_KEY=<the key above>. $0 idle, ~60-120s cold start on first request after idle (weights load from a volume-backed HF cache). See modal_app.py's module docstring for the full setup/teardown commands.

Every response also carries x-gpu-power-w / x-gpu-energy-j headers when NVML detects an NVIDIA GPU (power.py) — riprap's app/inference.py reads these for real measured power instead of a data-sheet estimate. No-ops on Mac Mini/MPS, where power_mac.py's local powermetrics sampling is the real signal instead.

Granite 4.1 (vLLM), also on Modal. modal_vllm_app.py deploys a separate app — different GPU profile (an 8B LLM at 8192 context wants more VRAM than the five specialists), different image (vLLM pins its own torch, incompatible with the specialist stack's). vllm_proxy.py fronts it with the same bearer-auth + NVML power pattern, since vLLM's own bundled server has no route for GPU power reporting and Modal exposes exactly one public port per Function.

modal secret create riprap-vllm-secret \
    RIPRAP_VLLM_API_KEY=$(openssl rand -hex 24) --env riprap
modal deploy modal_vllm_app.py --env riprap

Point Riprap at it with RIPRAP_LLM_PRIMARY=vllm, RIPRAP_LLM_BASE_URL=<url>/v1, RIPRAP_LLM_API_KEY=<the key above>. On a Mac Mini, Ollama serves this role instead (see below) — vLLM is Modal-only.

Native (recommended for local GPU/MPS access, e.g. a Mac Mini). Containers on Apple Silicon have no Metal passthrough — Docker/Colima on a Mac only gets CPU. Running server.py directly on the host is what gets real GPU acceleration on a Mac. This is how it runs on a Mac Mini: Ollama (the LLM) and this server both run natively, brew services / launchd for persistence across reboots. No auth by default — this path assumes a trusted local network; set RIPRAP_INFERENCE_API_KEY if you're exposing it beyond localhost.

# one-time: keep it running across logout/reboot
cat > ~/Library/LaunchAgents/com.riprap.inference.plist <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>Label</key><string>com.riprap.inference</string>
  <key>ProgramArguments</key>
  <array>
    <string>/path/to/riprap-inference/.venv/bin/python</string>
    <string>/path/to/riprap-inference/server.py</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
  <key>StandardOutPath</key><string>/tmp/riprap-inference.log</string>
  <key>StandardErrorPath</key><string>/tmp/riprap-inference.log</string>
</dict></plist>
EOF
launchctl load ~/Library/LaunchAgents/com.riprap.inference.plist

Docker (cloud / CUDA boxes). On a real NVIDIA GPU, containerizing is fine — CUDA passthrough works normally. A Dockerfile is not yet in this repo; pip install -r requirements.txt into any CUDA-enabled PyTorch base image and CMD ["python", "server.py"] is the whole recipe, since accelerator="auto" picks up CUDA the same way it picks up MPS.

Memory

Five models plus a two-model LLM (see the main repo) is a real budget on a 16 GB machine. server.py loads all five eagerly at startup so first requests aren't cold; if that's too much for your hardware, the easiest lever is lazy-loading per model (load on first predict() call instead of in setup()) — not yet implemented here.

License

Apache 2.0. See LICENSE and NOTICE for third-party model attribution.