AINode

Turn any NVIDIA GPU into a local AI platform.
Inference + fine-tuning in your browser. One container to install. Add nodes, they find each other.

ainode.dev  ·  docs  ·  Getting Started  ·  Screenshots  ·  What Works / What Doesn't

---

What AINode is

AINode is a self-hosted AI appliance for NVIDIA GB10 (DGX Spark, ASUS GX10) and any NVIDIA GPU box. It ships as one container that bundles:

• A modern web UI (chat, cluster topology, server console, downloads, training)
• An OpenAI-compatible API (/v1/chat/completions, /v1/completions, /v1/embeddings)
• A GB10-patched vLLM with Ray for cross-node tensor/pipeline parallel
• UDP node discovery for automatic clustering
• NFS-shared model storage so you download once and use everywhere
• Scripted fine-tuning (LoRA, QLoRA, full FT, DPO, distributed DDP)

One docker pull, one systemd unit per box, done. No host Python venv, no source-built vLLM, no fragile runtime wiring.

curl -fsSL https://ainode.dev/install | bash

---

Screenshots

Cluster view — 4 nodes, 487 GB aggregated VRAM

The "MASTER" node (head) runs the API and orchestrates. The smaller orbiting node (member) has its GPU reserved for a Ray worker that the head placed. The instance card shows DISTRIBUTED · TP=2 — the model is sharded across both GPUs.

Chat

Full-featured chat with streaming tokens, prompt history, code highlighting, per-message metrics (TTFT, tokens/sec, total tokens). Works against whatever model the cluster has loaded — solo or sharded.

Server — API console (LM Studio style)

Live developer console: which models are loaded on which node, OpenAI-/LM-Studio-/Anthropic-compatible endpoints, per-request logs with status codes and latency, eject-model buttons, copyable cURL snippets.

Downloads — live HF catalog

Browse trending HuggingFace models, with AVAILABLE / FITS GPU badges computed from your cluster's aggregate VRAM. Queue downloads to the shared NFS cache; any node can load them instantly.

Training — overview

Three quick-start paths: LoRA (lightweight, most users), Distributed DDP (multi-node fine-tuning), Full fine-tune (single large-memory node). Track active + completed runs, GPU-hours, and jump into dataset management.

Training — templates

Starter recipes for instruction tuning (Alpaca), chat fine-tuning (ShareGPT), classification heads, DPO / preference learning, and multi-node DDP. Each template ships a working dataset schema so you can start training in minutes.

Config — cluster

Pin the node's role (auto / master / worker), set a shared cluster_id so only matching nodes see each other, and inspect the current member list with per-node role, address, and last-seen.

---

Getting Started — step by step

Single node (solo mode)

1. Install Docker + NVIDIA container toolkit on your Linux box.
2. Pull the image and wire up the systemd unit:

   curl -fsSL https://ainode.dev/install | bash
   

   That one-liner:

   # resolves the highest numeric GHCR tag (never a floating :latest)
   docker pull ghcr.io/getainode/ainode:<latest-release>
   # pins it to ~/.ainode/image.env and installs a swappable systemd unit
   systemctl enable --now ainode.service
   

   The unit reads the pinned image from ~/.ainode/image.env (EnvironmentFile, Restart=always), so it survives cold power cycles and replays the models you had loaded on boot.
3. Open the UI at http://<your-ip>:3000. First-run onboarding walks you through picking a model. Click a model card → click Launch → chat.

Upgrade is ainode update (resolves + pulls the newest pinned release and restarts) — or ainode update 0.5.2 to pin a specific version.

Prefer to pull the image yourself? Both registries serve identical images — GHCR is canonical (what the installer uses), Docker Hub is a public mirror:

docker pull ghcr.io/getainode/ainode:latest      # canonical (always newest)
docker pull argentaios/ainode:latest             # Docker Hub mirror
# pin a release instead: …/ainode:0.5.2

Two nodes (distributed mode)

For models that don't fit on one GPU — e.g. a 70B-class model sharded across two DGX Sparks:

1. Wire a clean high-speed link between the two nodes (direct QSFP cable on its own /24, or a dedicated switch port). See Networking requirements — this matters.
2. Install AINode on both (step 1 above).
3. On the peer, set member mode in ~/.ainode/config.json:

   {
     "distributed_mode": "member",
     "cluster_interface": "enp1s0f0np0",
     "ssh_user": "sem"
   }
   

   sudo systemctl restart ainode.
4. On the head, set head mode and add passwordless SSH to the peer:

   {
     "distributed_mode": "head",
     "peer_ips": ["10.0.0.2"],
     "cluster_interface": "enp1s0f0np0",
     "ssh_user": "sem"
   }
   

   ssh-copy-id sem@10.0.0.2 && sudo systemctl restart ainode
   

5. Open the head UI — you should see both nodes, aggregated VRAM ("2 nodes · 244 GB · 2 GPUs"), and the instance badged as DISTRIBUTED · TP=2.

Want to do it from the browser instead? Open the Launch Instance panel, pick the model, set Minimum Nodes=2, click Tensor → LAUNCH. The UI writes the config and hot-swaps the engine for you.

---

Quantize a model (AWQ / NVFP4)

AINode can compress a full-precision model to 4-bit in the browser, on your own GPU — no external service. Open Training → Quantize a Model:

1. Base model — a Hugging Face repo id (Qwen/Qwen3.5-4B) or an installed model.
2. Scheme — AWQ (W4A16, proven on GB10 via awq_marlin) or NVFP4 (Blackwell-native 4-bit float).
3. Calibration samples — default 256 (from HuggingFaceH4/ultrachat_200k).
4. (optional) Push result to Hugging Face — requires a write token; pushes a private repo under your namespace.

The target node must be idle — quantization needs the full unified memory, so AINode refuses to start a quant job while a model is loaded (unload first, or pass force=true). The output lands in Installed as <org--name>-<scheme>, ready to serve.

AWQ is the proven path on GB10. NVFP4 quantization is newer; NVFP4 on multimodal models (e.g. Qwen3.5) is experimental and not yet verified — prefer AWQ for the Qwen3.5 family today.

Hugging Face tokens (read vs write). AINode keeps credentials in a local Secrets store (~/.ainode/secrets.json, mode 0600, obfuscated at rest) with two HF slots: a read token (download gated models) and a write token (push to the Hub — read-only tokens are rejected before any multi-GB transfer). Set them in Config → Secrets (each has a Test button showing the detected scope), or set the read token with ainode config --hf-token hf_xxx.

---

Features

---

Relation to the Community

AINode builds on excellent open-source work in the DGX Spark ecosystem. In particular, our base image inherits the patched NCCL from eugr/spark-vllm-docker (dgxspark-3node-ring branch), which we've found to be the most reliable variant for handling GB10 unified-memory topologies and fabric setups.

eugr's project remains the go-to for raw, high-performance vLLM clustering on Spark hardware. AINode layers a modern browser UI, one-command deployment, in-browser chat + OpenAI API, and distributed fine-tuning on top of that strong foundation.

Huge thanks to eugr and the contributors making multi-node Spark setups practical.

---

State of Distributed Inference (June 2026)

We owe readers the honest picture, not a checkmark-soup. Here's what's really running on our hardware.

What works today (verified)

• Single-node inference on any NVIDIA GB10 box (DGX Spark, ASUS GX10).

• Two-node tensor-parallel (TP=2) with one GPU per node on a direct-connect QSFP /24. Both GPUs show ~61 GB of ray::RayWorkerWrapper memory; NCCL chose NET/IB RoCE @ 200 Gb/s.

• Four-node cluster (3× DGX Spark + 1× ASUS GX10) — 487 GB aggregated VRAM, all four discovered automatically via UDP, topology visible in the browser UI. Verified April 2026.

• One-container-per-node install — curl -fsSL https://ainode.dev/install | bash -s -- --job worker installs in seconds with no model required.

• ainode role CLI sets master/worker/solo instantly.

• Worker nodes start immediately — no model download, no engine warmup. Web portal is up within seconds of systemctl start ainode.

• Shared model storage over NFS from an NVMe-oF-backed master.

• UDP cluster discovery on port 5679 with real peer-IP capture.

• Inference throughput: ~35 tok/s for a warmed-up model over the RoCE fabric.

• Four-node TP=4 — verified live on frontier MoE: nvidia/Qwen3-235B-A22B-NVFP4 served at TP=4 across 4× GB10 (~16–17 t/s single-stream, survived a 3,513-token prefill). The GB10 sm120 fix was --enforce-eager (vLLM's FlashInfer prefill kernel emits an illegal instruction under CUDA-graph capture on GB10).

• Federated serving — a master routes /v1/* to the node holding each model; models load/unload per node from the browser.

• Model stacking — multiple models per node, persisted and replayed on boot.

• In-browser quantization — AWQ and NVFP4 jobs run on an idle node and land the result in Installed (optionally pushed to Hugging Face).

What still needs care

• Ray over Tailscale — use physical cables or a dedicated switch.
• Single NIC per cluster subnet — multi-NIC ambiguity still breaks the NCCL ring.

Lessons learned the hard way

0. Role clarity eliminates half the problems. The single biggest UX improvement was ainode role master|worker|solo. Workers don't need a model, don't need to think, don't need config editing. They start in 3 seconds and announce themselves. The master is the only node that needs a model. Everything else follows from that.

1. Single NIC per cluster subnet. Multi-NIC ambiguity breaks NCCL ring setup silently; NCCL_SOCKET_IFNAME only tells NCCL which address to listen on, not which source the kernel picks for outbound traffic.

2. Ray placement groups outlive SIGKILL. Hung vLLM doesn't release the reservation; Ray's GCS still thinks the GPU is busy. Always docker rm -f the full chain before retrying.

3. Block-level shared storage is unsafe for multi-writer. NVMe-oF + ext4 mounted on two hosts corrupts under concurrent writes. Put NFS on top of a single-host mount.

4. The patched NCCL in eugr/spark-vllm-docker (dgxspark-3node-ring branch) is the only variant we've seen reliably handle GB10 unified-memory topologies. Our base image inherits it.

5. SSH from a root container into a host user fails silently when keys are mounted read-only from the host. Our entrypoint copies /host-ssh → /root/.ssh with correct perms and injects User <ssh_user> for peer IPs.

Why 3 nodes is harder than 2 (and why 4 is probably easier)

Two nodes: a single direct-connect cable on one /24. One cable, one subnet, one candidate interface per host. NCCL can't get confused. TP=2 splits the weights evenly. Solved problem.

Three nodes: no simple physical topology. Options:

• Triangle mesh (A↔B, B↔C, A↔C) with each link on its own /30 — community tooling assumes this, nobody autoconfigures it.
• Dedicated cluster switch with one NIC per node on an isolated subnet — easier, but a hardware purchase.
• Star topology — asymmetric latency, not recommended.

If your three nodes just share a regular LAN, you hit multi-NIC routing ambiguity (lesson #1). We did. NCCL ring setup succeeded; data never flowed.

Four nodes: paradoxically simpler once you commit to a switch, which is the only practical option for 4+. One NIC per node on a fresh /24, TP=4 lines up with vLLM's defaults, and the community has published recipes (eugr's recipes/4x-spark-cluster/, NVIDIA's internal 4× Spark reference setups).

Our hypothesis: the difficulty is not N nodes — it's how you wire N nodes. Two is forced (one cable). Three forces a topology decision. Four+ forces a switch, which is what the community tools expect. Stick to 2 now; buy the switch; jump straight to 4.

---

Networking requirements

AINode relies on NCCL for cross-node tensor-parallel, and NCCL works best when it owns a clean link.

• Passwordless SSH from the head's host user to every peer.
• Single active NIC per cluster subnet on every node. Multiple interfaces on the same /24 breaks the NCCL ring.
• No VPN between nodes for cluster traffic. Tailscale is fine for laptop→cluster SSH; not fine as the NCCL transport.
• Consistent MTU across the cluster subnet.

Recommended topologies

Diagnostic commands

# Confirm RDMA is live on your ConnectX-7
ibstat mlx5_0 | grep -E "State|Rate"         # expect "Active" + "Rate: 200"

# Confirm exactly one interface has an IP on the cluster subnet
ip -br -4 addr | grep 10.0.0                 # expect one line per node

# Confirm cross-node reach on the cluster subnet (not Tailscale)
ping -c 2 10.0.0.2
traceroute 10.0.0.2                          # 1 hop = right link

# Passwordless SSH works
ssh sem@10.0.0.2 true && echo OK

# After launching distributed: confirm NCCL uses RoCE, not Socket
docker exec vllm_node bash -c 'grep -E "Using network|NET/IB.*RoCE" \
  /tmp/ray/session_latest/logs/worker-*-01000000-*.out | head -5'
# Expect: "Using network IB" + "NET/IB ... mlx5_0:1/RoCE ... speed=200000"

Optional: GPU Direct RDMA (GDR)

Without GDR, traffic goes GPU → CPU → NIC → NIC → CPU → GPU. With GDR it bypasses the CPU hop. On GB10 the unified-memory CPU hop is cheap, so the win is smaller than on discrete GPUs but still measurable.

# Load the peermem module on each host (not the container)
sudo modprobe nvidia_peermem
echo nvidia_peermem | sudo tee -a /etc/modules-load.d/nvidia-peermem.conf

# Verify NCCL picks it up next launch
docker exec vllm_node bash -c 'grep "GPU Direct RDMA" \
  /tmp/ray/session_latest/logs/worker-*-01000000-*.out'
# Expect: "GPU Direct RDMA Enabled"

---

Shared model storage across a cluster

Downloading a 70 GB model three times on a three-node cluster is wasteful. AINode supports a shared models_dir so every node pulls from the same cache.

Block-level shared storage (NVMe-oF, iSCSI, Fibre Channel) is fast but unsafe for multiple Linux kernels writing simultaneously — ext4 / XFS have no distributed lock manager. Layer NFS on top:

  Storage array (NVMe-oF, SAN, local NVMe)
         │
         ▼
  MASTER NODE  ← ext4/XFS mounted here, owns the disk
    │   │
    │   └── NFS server exports /mnt/ai-models
    ▼
  WORKERS      ← mount the NFS share at /mnt/ai-shared

NFS over a 100G fabric gives 3–8 GB/s — vLLM model loading is a one-shot sequential read, so you won't notice. For 100 GB+ models where load time hurts, add an rsync-to-local staging step.

---

CLI reference

The installer puts a thin ainode wrapper at /usr/local/bin/ainode. Host-side commands (update) run directly; everything else is forwarded into the running container via docker exec. You never need to type docker yourself.

ainode update [version]      # resolve/pull newest (or pinned) tag + restart (upgrade in place)
ainode start                 # Start AINode (inference + web UI)
ainode stop                  # Stop AINode
ainode status                # Show cluster status
ainode models                # List available models
ainode service install       # Install the systemd unit
ainode service status        # Show systemd state + recent journal
ainode config                # Show current configuration
ainode logs -f               # Tail the engine log

Updating AINode

Releases ship through a tag-triggered pipeline: git tag vX.Y.Z → CI on a self-hosted Spark runner builds and pushes ghcr.io/getainode/ainode:X.Y.Z. To upgrade a node in place:

ainode update

That resolves the highest numeric GHCR tag (never a floating :latest), pulls it, pins it to ~/.ainode/image.env, and restarts the systemd service. Your config (~/.ainode/config.json), models (~/.ainode/models/), and fine-tune outputs are on the host — the container is stateless, so upgrades never touch your data.

To pin a specific version:

ainode update 0.5.2

To roll every node in a cluster from the master, use the Update all button or:

curl -X POST http://<master>:8000/api/cluster/update-all   # genuine pull + swap on every node

---

API

AINode exposes an OpenAI-compatible API. Drop it into any tool that speaks OpenAI:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
)

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

Works with Open WebUI, LiteLLM, LangChain, llama.cpp clients, and anything else that speaks OpenAI.

Metrics — /metrics (Prometheus) and /api/metrics (JSON)

AINode exposes its own metrics on the same port as the API:

curl http://localhost:8000/metrics           # Prometheus text exposition
curl http://localhost:8000/api/metrics       # JSON snapshot
curl http://localhost:8000/api/metrics/gpu   # GPU subset

Key series:

• ainode_uptime_seconds, ainode_build_info{version=...}
• ainode_requests_total, ainode_request_errors_total
• ainode_tokens_generated_total, ainode_tokens_per_second
• ainode_request_latency_milliseconds{quantile="0.5|0.95|0.99"}
• ainode_requests_by_model_total{model=...}
• ainode_gpu_utilization_percent, ainode_gpu_memory_used_bytes, ainode_gpu_temperature_celsius

Scrape config for Prometheus:

scrape_configs:
  - job_name: ainode
    static_configs:
      - targets: ["ainode-host:8000"]

---

Requirements

• OS: Ubuntu 22.04+ (DGX Spark OS works out of the box)
• GPU: NVIDIA GB10 (DGX Spark, ASUS GX10) — or any NVIDIA GPU with CUDA 13 drivers
• Memory: 8 GB+ GPU for small models, 128 GB recommended for the big ones, 240 GB+ aggregated for real sharded work
• Disk: 20 GB for the container image; more for models
• Docker: 24.0+ with the NVIDIA Container Toolkit

---

Why AINode?

---

Roadmap

• Core CLI + installer
• vLLM integration (patched NCCL for GB10)
• Web UI (chat, server, downloads, training, config)
• Multi-node auto-discovery + cluster topology view
• Automatic model sharding across nodes (TP=2 verified)
• NFS-shared model storage
• Unified container image + systemd install
• Browser-driven fine-tuning (LoRA / QLoRA / Full + DDP)
• Training artifact retrieval, LoRA merge, checkpoint resume
• Evaluation loop + W&B integration
• Prometheus metrics endpoint (/metrics)
• 4-node TP=4 sharded inference (verified — 235B-A22B-NVFP4)
• In-browser quantization (AWQ / NVFP4) + push to Hugging Face
• Federated multi-model serving (master routes /v1/* by model name)
• Model stacking (N models per node, persisted + replayed)
• Training + adapter-merge in a spawned GPU container (slim orchestrator)
• Deploy pipeline (tag → CI → GHCR → ainode update / cluster update-all)
• VLM (vision) serving with fp8-KV auto-skip on GB10
• AutoData — Δ-filtered synthetic-data generation (val-set lift objective)
• Model marketplace (custom registries)
• Mobile-friendly UI

---

Contributing

AINode is Apache-2.0 and welcomes contributions. See CONTRIBUTING.md — and please run the test suite (pytest tests/) before opening a PR.

---

License

Apache 2.0 — use it however you want.

---

_{crafted with ♥ by Jason Brashear · powered by argentos.ai}