llmserve

Local model residency bridge for vLLM. Single Rust binary with an optional TUI — opencode (or any OpenAI client) talks to this over OpenAI-compatible HTTP.

vLLM only. No Ollama API. No external model-registry daemon. vLLM has no catalog — building one is the point.

Stages

Delivered independently, in order:

Install

cargo install --path .

Usage

llmserve scan
llmserve reconcile          # human summary on stderr, JSON snapshot on stdout
llmserve reconcile | jq .   # pipe-clean JSON
llmserve clean              # dry-run cleanup plan
llmserve clean --apply      # delete planned non-vLLM / duplicates
llmserve status             # what's in memory + ready-to-load use ids
llmserve chat               # local coding agent (if exactly one model loaded)
llmserve use Qwen/Qwen3-8B  # load if needed, then agent session
llmserve use --no-chat      # only ensure loaded + print URL
llmserve unload 8002        # stop server by port / id / pid
llmserve diagnostics 8002  # crash-relevant runtime diagnostics

Coding agent tools (local only)

llmserve chat is a coding agent loop against your loaded vLLM model — no cloud:

Same workflow as other coding TUIs (search → read → edit → test), bound to local models only.

Security note: run_shell executes arbitrary local shell with no sandbox. File tools (read_file, write_file, str_replace, …) accept absolute paths and can reach anywhere the process user can. Prefer /readonly session mode when exploring untrusted prompts. Default session mode is write (prompt shows you✎>).

scan

Walks, in order:

1. ~/.cache/huggingface/hub/models--*/snapshots/*
2. extra_model_dirs from models.policy.toml
3. find-by-size sweep of $HOME as catch-all (≥1 MiB weight files)

A model dir = config.json + (*.safetensors | *.bin | *.gguf).

Parses config.json for architecture, dtype, num_hidden_layers, max_position_embeddings. Sums shard sizes for on-disk bytes.

Writes FACTS only to ~/.config/llmserve/models.facts.toml.
POLICY lives in models.policy.toml (hand-edited). Scan never overwrites it.

Missing models are tombstoned (absent_since timestamp), not deleted.

reconcile

Read-only observation:

1. Enumerate /proc/*/cmdline for vllm serve / api_server processes. Parse --model, --port, --gpu-memory-utilization, --max-model-len, --tool-call-parser, --served-model-name, --enable-sleep-mode. No port-scan.
2. Confirm each with GET /health and GET /v1/models.
3. Probe GET /metrics for in-flight/queued gauges. Pins actual metric names from the live build (vllm:num_requests_running / vllm:num_requests_waiting on current releases).
4. Tier every live server:

Sleep/wake is per-server: only when launched with --enable-sleep-mode (+ dev mode). Detected via GET /is_sleeping (404 = unavailable).

proxy (planned)

• ensure_awake(model): awake → use; asleep → POST /wake_up; cold → budget demotion then spawn
• Budget = sum of declared --gpu-memory-utilization vs MemAvailable from /proc/meminfo
• Tool-call smoke gate after every spawn and at adoption
• GET /state returns the same snapshot JSON
• Bind 127.0.0.1 only

Config

~/.config/llmserve/
  models.facts.toml     # written by scan
  models.policy.toml    # hand-edited; never overwritten
  owned.toml            # pids/ports we spawned (proxy)

models.policy.toml example

extra_model_dirs = ["/data/models"]
default_gpu_util = 0.45
default_max_len = 8192
# Optional fixed readiness deadline. Without this, llmserve automatically
# allows more time for larger weight sets. Set to 0 for no deadline.
# default_load_timeout_secs = 1800

[models."Qwen/Qwen2.5-Coder-1.5B-Instruct"]
gpu_util = 0.30
max_len = 8192
port = 8001
tool_parser = "hermes"
enforce_eager = false
# load_timeout_secs = 1800
extra_args = []

During a cold load, llmserve use prints a live heartbeat with elapsed time, available memory, and the latest vLLM log message. If vLLM exits or misses its deadline, llmserve prints the relevant log tail and stops the entire process group so partially loaded workers do not retain model memory. Interrupting a load with Ctrl-C performs the same cleanup before the command exits.

llmserve diagnostics [id|port] combines the binary and vLLM versions, active policy, process/PGID/RSS, host memory, GPU allocations, kernel GPU errors, and recent crash-relevant lines from the per-port vLLM log. Startup logs include the concrete PID and effective launch settings so failures can be correlated after several reloads.

Development

make build
make test
make clippy

Empirical notes (resolve before proxy design)

Target platform: GB10 DGX Spark style unified memory. Before implementing sleep rungs:

1. Measure MemAvailable before/after POST /sleep?level=1. If it does not move on unified memory, level 1 is not a usable rung.
2. Pin metric names from the installed vLLM build via reconcile → observed_metric_names.
3. Measure cold vs warm spawn with VLLM_CACHE_ROOT persistent, with and without --enforce-eager.

Budget double-count (spawn headroom)

Budget::can_fit currently compares declared_gpu_util_sum + need against MemAvailable / MemTotal. On unified-memory hosts a loaded model's allocation is already missing from MemAvailable and counted in declared_gpu_util_sum, so headroom is understated after the first load.

Measurement protocol (record on the target box before changing the formula):

Decision (pending numbers): either (a) compare need against 1.0 - resident_fraction using only measured availability, or (b) keep the declared sum but stop mixing it with MemAvailable (use 1.0 - declared_sum when declarations are trusted). Implementation lands in a follow-up commit with a budget_headroom_matches_measured fixture test once numbers are filled in.

License

MIT