nixos_host_inference_service

A small, NixOS-aware workstation runtime for local AI prototyping on AMD ROCm. It provides shared chat and text-to-speech APIs while centralizing GPU containers, models, and service lifecycle so consuming projects stay decoupled.

Purpose

This is a small workstation project for local prototyping, not a production AI platform. It keeps GPU setup, model downloads, containers, and engine lifecycle in one host service so consuming projects stay simple:

• applications depend on stable local HTTP contracts, not ROCm or Compose;
• several projects can reuse the same downloaded models and host setup;
• consumers can change models or later point at another compatible service without embedding inference infrastructure in each codebase.

The project runs the inference engines themselves. It is not a general AI gateway or request router: it does not select among cloud providers, load balance deployments, manage budgets, or provide multi-user authentication. Those capabilities can be added as a separate layer if local prototypes eventually need them.

Compared with LocalAI

LocalAI is the closest general-purpose open-source alternative: it can run chat and speech models behind one OpenAI-compatible API, acquire GGUF models, manage loaded backends, and target AMD ROCm. Its current model gallery also includes an F5-TTS implementation.

This project deliberately uses smaller, proven components instead:

• Ollama is already verified for chat and GGUF inference on this host GPU;
• the dedicated F5 service is verified on ROCm and implements this project's paragraph chunking, silence trimming, crossfades, and narration pauses;
• explicit service switching makes the one-heavy-workload policy easy to see and debug on a ~16 GB GPU;
• the repository avoids LocalAI's broader backend, UI, agent, authentication, and multi-user platform surface.

LocalAI may become the simpler choice if it proves the same RDNA4 GPU behavior, F5 quality, and long-form narration semantics. It has not been run on this host, so this is a comparison of documented capabilities, not a local benchmark.

Audience and prerequisites

This project assumes intermediate NixOS knowledge. You should already be comfortable editing and rebuilding your NixOS configuration, using flakes and nix develop, running Docker Compose, and diagnosing basic container or device permission problems.

The host must provide:

• an AMD GPU supported by ROCm;
• Docker Engine with Compose, usable by your account;
• /dev/kfd and /dev/dri available for container passthrough.

This repository does not configure the host or provide a NixOS module. The supported runtime is Docker Compose (COMPOSE_CMD can select a wrapper). nix develop supplies the user-space tools used by the recipes: just, Docker CLI, curl, jq, and the Hugging Face CLI.

What works today

Both services listen on loopback and are intended for applications on this machine. They are separate GPU workloads and API endpoints.

Chat

Verify the installation

nix develop
MODEL=smollm2:135m just smoke

just smoke starts Ollama, waits for readiness, pulls the selected model if it is missing, makes an OpenAI-compatible chat request, and verifies that Ollama reports GPU use. It leaves Ollama running. The default smoke model is smollm2:135m; use MODEL=tinyllama just smoke, for example, to test another model.

Run the smoke test for initial setup, after GPU or container configuration changes, or while troubleshooting. It is an acceptance check, not the normal service launcher.

Use chat from another project

nix develop
just start
just wait-ollama

The consuming project needs the endpoint:

OPENAI_BASE_URL=http://127.0.0.1:11434/v1

The consumer selects an already-installed model in each Chat Completions request. MODEL configures just recipes; it does not set a server-wide model for consumers. Install a new Ollama-library model with:

MODEL=tinyllama just pull

Ollama does not authenticate this host-local endpoint, making the Authorization header in the curl example purely illustrative for compatibility reasons:

curl --silent --show-error --fail \
  "$OPENAI_BASE_URL/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer any-key-works" \
  -d '{
    "model": "smollm2:135m",
    "messages": [{"role": "user", "content": "Reply with only: OK"}],
    "stream": false
  }'

Stop chat when it is no longer needed with just stop.

Text-to-speech

F5-TTS clones a fixed local reference clip. Before making a synthesis request, provide:

state/tts/ref/basic_ref_en.wav

The clip's spoken transcript must be: Some call me nature, others call me mother nature. This is a local fixed reference, not a voice upload or voice-library API.

Verify the installation

nix develop
just start-tts
just wait-tts
just tts-smoke

The first start builds the image and may download model weights, so it can take several minutes. wait-tts waits for the HTTP health endpoint. The smoke test sends one short sentence and writes state/tts/out/smoke.wav; listen to that file to complete the check.

Use TTS from another project

nix develop
just start-tts
just wait-tts

Configure the consumer with:

TTS_BASE_URL=http://127.0.0.1:8020

Example request:

curl --silent --show-error --fail --max-time 120 \
  "$TTS_BASE_URL/v1/audio/speech" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer any-key-works" \
  -d '{
    "model": "f5-tts",
    "input": "Hello from a local application.",
    "voice": "default",
    "response_format": "wav"
  }' \
  --output speech.wav

GET /health is a quick readiness check:

curl --silent --show-error --fail "$TTS_BASE_URL/health"

It returns HTTP 200 and JSON describing the loaded engine and model when the server has initialized successfully. It does not synthesize speech or prove GPU inference; use just tts-smoke for that end-to-end check.

Speech generation is synchronous. Clients should allow a timeout of several minutes for long input; consumer applications configure their own HTTP timeout.

Stop TTS when it is no longer needed with just stop-tts.

Switching services

Only one GPU-heavy service should be resident on the ~16 GB GPU. start-tts stops Ollama automatically. To switch back to chat:

just stop-tts
just start
just wait-ollama

Optional Hub token (HF_TOKEN) for rate limits / gated models — see env.example. Do not commit real tokens.

Model management

Use just pull for a model published in the Ollama library. Use the GGUF path when you need to download a compatible quantized model file from Hugging Face and register it with Ollama:

MODEL=tinyllama just pull

just hf-download bartowski/SmolLM2-135M-Instruct-GGUF SmolLM2-135M-Instruct-Q4_K_M.gguf
just import-gguf state/huggingface/bartowski/SmolLM2-135M-Instruct-GGUF/SmolLM2-135M-Instruct-Q4_K_M.gguf smollm2-hf-q4
MODEL=smollm2-hf-q4 just smoke

Limits

• One GPU-heavy job at a time on ~16 GB VRAM. Stop chat before TTS (and the reverse).
• Host-local only — published on loopback, not LAN.
• TTS is functional but not expressive. F5 produces intelligible, mostly flat English narration. Comparative results are in research/tts/evaluation.md.
• No voice-cloning library / upload API yet; F5 uses a fixed reference wav.
• No streaming TTS; sync HTTP with a long client timeout.
• Models and caches live under gitignored ./state/ (bind-mounted). Disk can grow large.

Data on disk

These directories are bind-mounted into the containers. just down does not delete them.

Command reference

just list           # list all recipes
just start          # create/start Ollama
just stop           # stop Compose services without removing containers
just down           # remove Compose containers and networks; keep state/
just pull           # pull MODEL from the Ollama library
just models         # list Ollama models
just wait-ollama     # wait until the chat API is ready
just smoke          # chat API and GPU acceptance check
just ollama-logs

just hf-download …  # download named Hugging Face files into state/
just import-gguf …  # register a downloaded GGUF with Ollama

just start-tts
just wait-tts
just tts-smoke      # short one-sentence HTTP check → state/tts/out/smoke.wav
just stop-tts
just tts-logs

AI-assisted development

This repository was developed with substantial use of AI coding assistants in Cursor. AI helped with planning, research, implementation, debugging, and documentation; much of the code and prose was drafted or revised with that assistance. The human maintainer chose the scope and architecture, reviewed the changes, and ran the documented acceptance tests on the target NixOS + AMD hardware. Responsibility for the resulting code, tests, and licensing remains with the human maintainer.

Project notes

• Design and agent brief: AGENTS.md
• Dated implementation notes: logbook.md
• TTS research, fixtures, and evaluations: research/tts/
• Deferred TradingAgents migration: docs/tradingagents-migration.md