sparkDash ⚡ — Multi-unit monitoring dashboard for NVIDIA DGX Spark

_{by Mia'a AI Lab}

_{macOS (darwin) SSH collectors authored by C. Michael Gibbs and Nyx Voss (Onyx AI Labs) — commits. sparkDash by Mia's AI Lab.}

sparkDash is a real-time web dashboard for one or more NVIDIA DGX Spark (GB10) machines in a single browser window. It streams GPU, CPU, unified memory, storage, network, and local LLM metrics — and lets you add, edit, reorder, or remove Sparks from the UI without restarts or code changes.

It also supports non-Spark units: any Linux machine with an NVIDIA GPU (e.g. a workstation with a dedicated RTX/L-series card) can be added as a dedicated GPU host and monitored the same way via SSH and nvidia-smi. For these units the dashboard correctly separates RAM (system memory) from VRAM (discrete GPU memory).

LLM Prompt Showcase

_{Download MP4 · also in assets/llm-showcase.mp4}

---

Table of contents

• Latest version changelog
• Features
• ComfyUI monitoring
• Hermes Agent monitoring
• Tailnet monitoring
• Full changelog
• Quick start
• Architecture
• Tech stack
• Repository layout
• REST API
• Configuration
• Security
• Scripts
• How it works
• Contributing
• License

---

Latest version changelog

Version 1.8.9 — TensorFold backend

• TensorFold (ashhart/TensorFold) is detected from /v1/models (owned_by: tensorfold) and labeled on the LLM card and Overview. Live tok/s reads cumulative token totals from /health when the server publishes them; stock TensorFold does not yet, so it shows 0 tok/s until it does. Benches and the showcase work as on any OpenAI-compatible server.
• q27 backend, custom prefill size, remote-Spark benches over an SSH tunnel, an on-demand Remote bench host, hide worker nodes, and a share-as-image card for bench results.
• Fixes for the decode-bench request quota and 24×/32× budget, long prefills dying at ~5 min, SGLang prefill latching, SPARKDASH_TOKEN in compose, Tailscale address classification, and remote SSH session churn.

Full history: CHANGELOG.md

---

Features

---

ComfyUI monitoring

sparkDash can optionally monitor a ComfyUI instance on each Spark — the same way it probes local LLMs, but focused on jobs and queue, not a second copy of GPU/RAM bars (those stay on the GPU / CPU panels).

What is supported

Not claimed: true per-job VRAM (Comfy does not expose that cleanly over HTTP). Host GPU/VRAM remains on the GPU panel. Live step progress depends on Comfy broadcasting WS events; stock Comfy often scopes detailed progress to the client that submitted the prompt.

How to enable (per Spark)

1. Open the Spark tab → Edit (pencil).
2. Enable ComfyUI monitoring.
3. Set port if needed (default 8188).
4. Save.

The Spark page Services section shows the ComfyUI card. On Overview, a small Comfy chip appears for that unit.

Connectivity Test (in Edit) includes ComfyUI when monitoring is enabled.

ComfyUI side requirements

• ComfyUI must be reachable from the sparkDash server on the probe host:
  • Local Spark (isLocal): sparkDash probes 127.0.0.1:{port} (use Docker network_mode: host if the dashboard runs in a container).
  • Remote Spark: probe uses the Spark LAN IP (same as LLM probes).
• For Open from another machine’s browser, Comfy should listen on a reachable interface (e.g. --listen 0.0.0.0), not only loopback, and the Spark’s LAN IP must be set correctly in Edit.

Config fields (persisted on the Spark)

Related API

Env (optional): COMFY_PORT (default 8188), COMFY_PROBE_TIMEOUT_MS, POLL_INTERVAL_COMFY.

---

Hermes Agent monitoring

sparkDash can optionally monitor Hermes Agent (nousresearch/hermes-agent) on each unit and run one-click updates for you over SSH.

What is supported

How to enable (per Spark)

1. Open the Spark tab → Edit (pencil).
2. Enable Hermes Agent.
3. Save — background checks start immediately.

The Update Hermes button appears in the Spark header/mobile action row; it turns warning-yellow with a commit-count badge only when an update is actually available. It also appears on Overview (batch) when at least one unit has Hermes enabled.

Connectivity check note: local units run the check as the host user (via setpriv/nsenter, never as container root); remote units run it over SSH. Either way, the logged-in user needs permission to read the Hermes repo.

Side requirements

• Hermes Agent must be installed on the target machine — sparkDash only checks & updates; it does not install it. The binary is looked up in ~/.local/bin and /usr/local/bin.
• SSH user must be able to run hermes update --check / hermes update non-interactively (key auth recommended).
• An update can take a few minutes (repo pull + dependency reinstall); a stale *.lock file from a crashed run is cleared before each attempt.

Config fields (persisted on the Spark)

Related API

Env (optional): POLL_INTERVAL_HERMES (default 600000 ms), HERMES_UPDATE_TIMEOUT_MS (default 600000 ms).

---

Tailnet monitoring

Opt-in per unit (default off). Runs tailscale status --json on the host and shows a Tailnet card under Resources.

This closes a blind spot every LAN-based check shares, including sparkDash's own SSH liveness. When tailscaled loses its session with the coordination server, SSH/GPU/LLM can all stay healthy while the box is unreachable from off-LAN.

What is supported

Asked of each node about itself. Peer state is never the verdict. The probe is read-only (tailscale up / down / login are never run).

How to enable

1. Open Edit Spark.
2. Tick Tailnet monitoring.
3. Save. The Tailnet card appears under Resources.

Host requirements

• tailscale CLI on the monitored host, and tailscaled running.
• Remote units: existing SSH. Local Docker: nsenter into the host mount namespace (same as nvidia-smi; /host/proc is already bind-mounted).

Config fields

Env (optional): POLL_INTERVAL_TAILSCALE (default 30000), TAILSCALE_PROBE_TIMEOUT_MS (default 8000).

---

Quick start

git clone https://github.com/MiaAI-Lab/sparkDash.git
cd sparkDash

# Production (Docker; loopback-only by default)
docker compose up --build -d

# Or development (host, with hot reload)
npm install
npm run dev

• Docker: open http://127.0.0.1:5555 on the host (arm64 image, auto-restart, host mounts for GPU/metrics access)
• Dev: Vite on http://localhost:5173 (proxies API/WS to Express)

For another computer, keep the server on loopback and use an SSH tunnel:

ssh -N -L 5555:127.0.0.1:5555 user@sparkdash-host

Then open http://127.0.0.1:5555 on that computer. For shared access, use an authenticated TLS reverse proxy, Tailscale Serve, or set BIND_HOST=0.0.0.0 and SPARKDASH_TOKEN. A direct LAN bind without a token is open by default: anyone who can reach the port can change settings and power units off, and the header shows an Open access warning. Set SPARKDASH_TOKEN to require a token, or SPARKDASH_ALLOW_OPEN_REMOTE=0 to make a tokenless LAN bind refuse to start.

When the server has SPARKDASH_TOKEN set, the dashboard asks for it: the first request or live-telemetry connection the server turns away opens an Access token dialog. Enter the token once; it is checked against the server, stored in this browser only, and the live connection reconnects with it — no reload, no devtools. Settings → Access token shows whether one is stored and lets you change or clear it.

For development with Docker (source-mounted, HMR):

docker compose -f docker-compose.dev.yml up --build

Remote units + SSH keys (Docker): SSH is executed inside the container on the sparkDash host (typically the head DGX). Configured LAN IPs are from that host’s point of view, not your laptop. OpenSSH looks for keys under /root/.ssh in the container — the host user’s ~/.ssh is not used unless you bind-mount it. Uncomment this volume in docker-compose.yml (and recreate the container):

- ${HOME}/.ssh/id_ed25519:/root/.ssh/id_ed25519:ro

If the key file has a non-default name (e.g. id_ed25519_shared), mount it as id_ed25519, or set SSH_IDENTITY_FILE to the path inside the container. Keep the file mode 600. The unit that runs sparkDash itself should be added with This host (local collectors — no SSH for metrics).

---

Architecture

Design principle: one Spark model, N instances. Every unit is a record in config/sparks.json with a kind field (spark or host). The same SparkMonitor, SystemCollector, and LlmProbe code runs for all of them. Adding a unit is a config change, not a code change.

┌────────────────────── Docker container (sparkDash) ────────────────────────┐
│  Express (server/)                                                         │
│  ├─ config/sparks.json        Spark registry (API read/write)              │
│  ├─ SparkRegistry             load/persist Sparks; change events           │
│  ├─ SparkMonitor (per Spark)  collector + LLM probe + rate baselines       │
│  │   ├─ SystemCollector       local sysfs/proc OR remote SSH               │
│  │   └─ LlmProbe              HTTP to host:LLM_PORT, backend autodetect    │
│  ├─ REST /api/*                                                            │
│  └─ WebSocket /ws             snapshot stream to browsers                  │
│  React SPA (src/)  — Overview + per-Spark pages, themes, dialogs           │
└────────────────────────────────────────────────────────────────────────────┘
         │ SSH (key or sshpass)                    │ HTTP :8888
         ▼                                         ▼
    remote Spark(s)                         each Spark’s LLM server

Data flow

Browser  ←→  WebSocket /ws   ←→  SparkMonitor.snapshot()  ←→  collectors
Browser  ←→  REST /api/*     ←→  SparkRegistry + SparkMonitor

Poll loops run in the background (even with no clients) so rate metrics — tokens/s, network bytes/s, disk I/O — stay correct.

---

Tech stack

---

Repository layout

sparkDash/
├── src/                 React + TypeScript SPA
│   ├── api/             REST client + shared types
│   ├── components/      Overview, Spark pages, dialogs, UI primitives
│   ├── hooks/           WebSocket snapshot, routing
│   └── theme / CSS      Tailwind v4 + four themes
├── server/              Express + WebSocket (plain JS ESM)
│   ├── sparks/          SparkRegistry, SparkMonitor
│   ├── collectors/      SystemCollector, LlmProbe, ssh
│   ├── secretsStore.js  Encrypted password persistence
│   └── validate.js      Host/user validation (SSRF-minded)
├── config/              Runtime state (volume; secrets gitignored)
├── assets/              Screenshots
├── Dockerfile           Production multi-stage arm64
├── docker-compose.yml   Production
├── docker-compose.dev.yml
└── deploy.sh            Rebuild / recreate helpers

---

REST API

There is no application authentication on the HTTP/WebSocket API. sparkDash therefore binds to loopback and refuses direct LAN binding. Use an SSH tunnel, authenticated TLS reverse proxy, or Tailscale Serve; see Remote access.

/api/fleet-energy samples the configured fleet independently every two seconds. It estimates each node as GPU board draw + a CPU utilization model (5.2–65 W) + 23 W of memory/network/base overhead, clamped to the DGX Spark power envelope. Current and hourly fleet watts require fresh, simultaneous telemetry from every node; coverage fields make gaps explicit. Minute buckets are persisted at mode 0600 for rolling 24-hour and 31-day windows. Wh/output-token is reported when exactly one configured node has role head and exposes a monotonic LLM output-token counter. These values are estimates, not wall-meter measurements. Restart sparkDash after changing fleet membership so the persisted series has one stable node set.

---

Configuration

Global settings (UI or API)

Gear icon in the header, or GET/PUT /api/settings:

Environment variables

Copy .env.example to .env if needed:

For compatibility, SSH_CONTROL_PERSIST is accepted as a seconds-based fallback when SSH_CONTROL_PERSIST_SECONDS is unset. The existing SSH_MULTIPLEX=0 switch also disables reuse. SSH tunnels always use an independent connection.

The listener and both Compose files default to 127.0.0.1. Existing Docker users who opened http://<host-ip>:5555 must migrate to an SSH tunnel, authenticated reverse proxy, Tailscale Serve, or BIND_HOST=0.0.0.0 SPARKDASH_TOKEN=... (without the token a 0.0.0.0 bind is open to the network unless SPARKDASH_ALLOW_OPEN_REMOTE=0). Recovery: BIND_HOST=127.0.0.1 docker compose up -d --force-recreate.

Adding a unit

1. Open the + tab.
2. Choose Unit type:
   • NVIDIA DGX Spark — the default; hardware summary shows DGX Spark specs and the CX7 IP field is available.
   • Dedicated GPU host — any Linux machine with an NVIDIA GPU. It is monitored exactly like a Spark (SSH + nvidia-smi) but is not reported as a DGX Spark: the header shows a detected hardware summary (GPU model, CPU, RAM) instead of fixed GB10 specs, and the page shows separate RAM and VRAM panels (VRAM from nvidia-smi, RAM from system memory). On the unit page, RAM → Network → Storage stack in the right column with GPU filling the left column. A host with more than one GPU needs nothing extra: every card nvidia-smi lists is collected, the header names them all, the GPU panel shows a block per card, and metrics.gpu stays the aggregate (hottest / busiest card, summed power and VRAM) with the per-card detail under gpu.gpus[].
3. Set Name and choose whether this is This host. Local units do not require a LAN IP or SSH; their optional LAN IP enables browser links and directed Wake-on-LAN. Remote units require a LAN IP/host, SSH user, and key or password. Key auth in Docker needs a key mounted into the container (see Quick start).
4. Test shows pass/fail/skipped for host collectors/SSH and each enabled service (LLM, ComfyUI, Hermes Agent, Tailnet). Every enabled capability must pass; disable an unavailable optional service before saving if it should not be monitored.
5. Save — a tab appears and metrics start streaming.

Power controls (shutdown / Wake-on-LAN)

• Shutdown (per Spark or Shutdown All on Overview) runs the host helper /usr/local/bin/spark-shutdown with passwordless sudo. The helper contract is two invocations:

  Invocation	Expected behaviour
  spark-shutdown	Schedule the graceful shutdown
  spark-shutdown --check	Print an acknowledgement, exit 0, change nothing

  --check is what proves authorization before anything is scheduled, so a sudoers rule scoped to the helper is enough:

  sparky ALL=(root) NOPASSWD: /usr/local/bin/spark-shutdown
  

  A helper without --check still works when sudo is granted more broadly (the authorization probe falls back to sudo -n true), but a rule limited to the helper path needs --check support.

• On a local unit, the helper runs on the Spark itself. When the dashboard is in Docker that means the invocation first enters the host mount namespace (nsenter --mount=/host/proc/1/ns/mnt -- sudo -n …), using the same HOST_PROC_PATH mount and privileged: true the collectors already need. A bare-host install calls sudo directly. The helper always resolves against the host filesystem, so it does not need to exist inside the container.

• Wake / Wake All send a UDP magic packet (port 9). The MAC is taken from the enP7s7 interface automatically while the Spark is online (persisted as detectedMacAddress). Optionally set a MAC override in Edit Spark. Broadcast is derived as /24 from LAN IP, or 255.255.255.255 if LAN IP is missing.

• Batch shutdown only targets online Sparks; offline nodes are skipped.

• Power APIs are mutations: with SPARKDASH_TOKEN set they require it; without it they are open on loopback (local trust) and on a remote bind, unless SPARKDASH_ALLOW_OPEN_REMOTE=0 makes that bind fail closed.

Themes

Header theme control cycles:

Choice is stored in localStorage.

---

Security

• SSH passwords are not stored in sparks.json and are never returned by the API.
• Passwords are encrypted with AES-256-GCM in config/sparks-secrets.json (survives restarts).
• Encryption key: config/.secrets-key (auto-generated) or SPARKDASH_SECRETS_KEY. Do not delete the key file or encrypted secrets become unreadable.
• Target validation rejects clearly unsafe IPv4 targets (link-local 169.254.0.0/16, 0.0.0.0/8, multicast/reserved ≥ 224). Private, loopback, and public addresses are allowed so LAN and remote Sparks work.
• SSH and HTTP probes use short timeouts (about 5 s SSH connect, 3 s HTTP) so a hung host cannot stall the poll loop.
• Prefer SSH keys over passwords. In Docker, mount the private key into /root/.ssh (see Quick start); passwords are the only SSH secret the app stores itself.
• Loopback installs remain local-trust. A remote bind (BIND_HOST not loopback) without SPARKDASH_TOKEN is open by default: anyone who can reach the port can read telemetry, change settings and power units off, and the header shows an Open access warning (dismissible per browser). Set SPARKDASH_TOKEN to require a bearer token for mutations and remote telemetry/WebSocket, and SPARKDASH_ALLOW_OPEN_REMOTE=0 to refuse to start a remote bind without one. GET /api/health reports which applies as authMode: loopback-open, bearer, open-remote, or required-missing.
• One-off remote benchmark hosts must be listed in SPARKDASH_BENCH_HOSTS.
• Tested operator capacity for this remediation: 12 units.

---

Scripts

---

How it works

Local vs remote Sparks

One SystemCollector path for both modes. When spark.isLocal is true, metrics come from host sysfs/proc and nvidia-smi (often via nsenter into the host namespace). Remote Sparks wrap the same commands in a shared sshExec() helper (key agent or sshpass). The helper reuses an authenticated OpenSSH transport by default so frequent metric polls do not create a new SSH/PAM login lifecycle each time. Set SSH_CONTROL_PERSIST_SECONDS=0 to disable reuse. For kind: "host" units, actual hardware (GPU model, driver version, CPU, RAM) is detected once and cached in place of the static DGX Spark specs, and GPU VRAM comes straight from nvidia-smi while system RAM is read from /proc/meminfo.

Remote SSH sessions and host memory

Older sparkDash versions could create hundreds of SSH/PAM login sessions per minute on each remote host. Issue #73 documents the resulting session churn and observed polkitd memory growth. Connection reuse reduces this churn while retaining the collector refresh cadence. After updating, verify that metrics keep advancing and that new SSH authentications/PAM session opens fall after the initial connection; a new SSH client process for each collector command is still expected.

If host memory remains low, compare Linux MemAvailable and per-process resident/swap usage. Memory retained by polkitd requires separate OS investigation: polkit PR #653 fixes a reference leak in NoNewPrivileges queries. Check whether your distribution's polkit package includes that fix. SSH reuse neither applies the OS patch nor releases memory already retained by another process.

Graceful degradation

Collectors catch errors and return zero/default metrics instead of crashing the loop. After sustained liveness failures, a Spark is marked offline; the UI shows stale or empty states rather than hard errors.

Hot configuration

Name, IP, SSH credentials, LLM port, and device/interface filters update the running SparkMonitor without tearing down poll loops or losing rate baselines. Registry writes are atomic (temp file + rename).

LLM probe

Each configured LLM port gets its own LlmProbe instance running in parallel. Probes auto-detect backends:

• llama.cpp — /slots for live decode rates; model from /props
• ds4-server (Entrpi/ds4-on-spark) — /v1/models (owned_by: ds4.c) + Prometheus ds4_* token counters for live tok/s
• EXL3 (ExLlamaV3 tools/serve_openai.py) — /v1/models (owned_by: exl3) or /health {ok, busy}; live tok/s from /health cumulative counters
• q27 (signalnine/q27 engine) — /v1/models (owned_by: q27) or Prometheus q27_* series; live tok/s from q27_*_processed counter diffs (completion-based totals as fallback), exact computed-only prefill with the cached/uncached split doubling as the prefix-cache hit rate, TTFT/E2E/ITL p95 histograms, and constant-0 preemptions (FIFO admission, no wait queue)
• TensorFold (ashhart/TensorFold) — /v1/models (owned_by: tensorfold). It has no /metrics, and the CUDA server's /health is just {ok: true}, so live tok/s appears only when /health publishes cumulative prompt_tokens_total / completion_tokens_total (same contract as EXL3); otherwise the card shows the model and 0 tok/s. Decode/prefill benches and the showcase work regardless.
• vLLM / sglang — /v1/models; sglang via /server_info (last_gen_throughput when metrics off; /get_server_info fallback), vLLM via Prometheus /metrics counters (scientific notation supported)

Rates are derived from per-probe cumulative counter diffs (or SGLang sticky throughput while it moves). Multiple ports can be added or removed at runtime without restarting the monitor.

Live probes still use the LAN IP on remote units. Decode and prefill benches try that same HTTP target first; if it is closed they open an SSH local-forward onto the remote’s 127.0.0.1 so loopback-bound servers (ds4 start.sh default) can still be measured. The tunnel is torn down when the job finishes or is cancelled.

---

Contributing

Contributions are welcome. Conventions:

• Server: plain JavaScript ESM
• Client: TypeScript + React
• Prefer extending the shared Spark model over per-unit special cases

---

License

MIT — Copyright (c) 2026 Mia'a AI Lab

---

Acknowledgements

• Built for the NVIDIA DGX Spark (GB10) on ARM64
• Rebuilt from a legacy multi-unit dashboard with a single shared Spark model (no copy-pasted “Spark N” code paths)
• LLM probe behavior refined from production monitoring experience