NeuDecide: browser demo

Speech → tool call, entirely client-side: an exported NeuDecide checkpoint (export/) running on onnxruntime-web (WASM, multi-threaded when the page is cross-origin isolated). Based on live.py. Pick a scenario or build your own tool list, then record from the mic, upload a file or click a sample clip.

Run locally

# 1. export a checkpoint (see export/README.md) -> export/out/<name>/
# 2. serve the demo; export/out is mounted at /models/
python demo/serve.py            # http://127.0.0.1:8000

• ?model=models/<name>/ picks another export (default model/, the bundled model). Any URL works, e.g. a Hugging Face model repo's https://huggingface.co/<repo>/resolve/main/.
• The model always runs the q4 variant (int4 weights and kernels, 41 MiB download); exports must include it.
• The mic needs a secure context. localhost counts; on a remote box use ssh -L 8000:localhost:8000 <host>.

How it works

• js/worker.js loads the model into a module worker, so the UI never blocks.
• js/neudecide.js is the pipeline, the same as export/runtime.py: audio_encoder once, tool_encoder once per tool list, then greedy decoder_step with a KV cache.
• js/tokenizer.js has SentencePiece BPE plus Python-exact json.dumps. The model has to see the tool list serialized byte-for-byte as in training.
• js/audio.js handles mic capture (AudioWorklet, raw PCM) and file decoding. Both are resampled to 16 kHz mono.
• Tools use the JSON-Schema format the synthetic training data uses (name, description, parameters{type, properties, required}), not live.py's Audio2Tool dialect. The shipped scenarios are in scenarios.json. Edits are saved in localStorage, per scenario.

Tests

export/.venv/bin/python demo/tests/make_fixtures.py --model-dir export/out/<name>   # Python reference
cd demo/tests && npm install && npx playwright install chromium
node run_tests.mjs          # tokenizer + json.dumps parity, full decodes on onnxruntime-web (Node)
python ../serve.py --port 8765 &  node e2e.mjs http://127.0.0.1:8765/   # headless browser, fake mic
python ../serve.py --port 8766 --hf &  node e2e.mjs http://127.0.0.1:8766/   # as HF Spaces serves it

Deploy to Hugging Face Spaces

HF_TOKEN=hf_xxx export/.venv/bin/python demo/deploy_space.py --repo neuphonic/neudecide --private

This deploys a Docker Space by default. The Dockerfile runs serve.py --no-isolation on port 7860, and the q4 model ships under model/. A private static Space can't serve the page's own scripts or model files to the browser (they come back 401), but a private Docker Space's requests go through HF's authenticated proxy. The model runs single-threaded there: embedded on huggingface.co the page can't be cross-origin isolated, so the container doesn't send COOP/COEP. --sdk static still deploys the static variant, which only works for a public Space. serve.py --hf reproduces static-Space serving locally.

Device lab (lab.html)

The NeuDecide diagram, live: Audio → Model → { } → four devices (robot vacuum, robot arm, smartwatch, lamp) drawn as pixel art. Click a device to zoom in and see its choices; hold space (or the Record button), upload a clip, or play a sample, and watch the choices light up as the model weighs them, the chosen one glow, its JSON stream in, and the device act it out. With no device open, the model gets all 23 choices and picks the device itself; the page zooms to it.

The vacuum uses the demo's built-in tools unchanged (so the sample clips work); the arm, watch and lamp have six argument-free choices each (js/lab/devices.js). Tested with the real model on the three sample clips: all routed correctly, both from the overview and inside the vacuum.

Mac control (local only)

Voice control for four apps -- Granola, Slack, Chrome and Notion -- with 10 functions each, plus openApp and goBack everywhere.

pip install pynput sounddevice numpy
python serve.py --mac --hotkey    # then keep http://127.0.0.1:8000 open in a tab

Hold ⌃⌥ in any app, speak, release (or tap ⌃⌥ for hands-free, tap again to stop). The page uses the tools of whichever of the four apps is in front; in any other app, the home list (openApp). Results arrive as macOS notifications. Without --hotkey, use the page's Record button.

• Granola needs a personal API key: in the Granola app, Settings > API > generate one (Business or Enterprise plan), then export GRANOLA_API_KEY=grn_... before starting serve.py. Only notes Granola has finished summarising are available.
• Granola calls without a person or topic use the current meeting (the last one opened, summarised or searched), else the latest.
• Slack never sends on its own: drafts wait for sendMessage ("send it").
• Action items, questions and the follow-up email need a language model (summaries use Granola's own): Ollama running locally (NEUDECIDE_LLM=ollama, OLLAMA_MODEL), or ANTHROPIC_API_KEY (NEUDECIDE_LLM=claude).
• Chrome's summarizePage needs View > Developer > Allow JavaScript from Apple Events.
• Permissions for the app running serve.py: Accessibility, Automation (asked per app), Microphone and Input Monitoring.
• The tools live in scenarios.json (mac_control/<app>), the code in mac_actions.py (ACTIONS).