3D LLM Sandbox

Meet a tiny robot with a big attitude. Hold a button, tell her "build me a tower", and she does: blind, never seen a thing, absolutely sure she just made a masterpiece. Wrong height, wrong place, zero doubt, plenty of sass. She lives entirely on your own machine: local model, local ears, local voice, no cloud. Come argue with her.

https://github.com/user-attachments/assets/aae3e235-0117-4fcf-9fe7-6f158a88ef0f

What this experiment is about

This is not a game. It is a test of one idea: split the controls by how fast your intent is.

• Walking, looking, aiming: your hands already know how to do that. Keyboard and mouse, under 16 ms, the model never gets involved.
• "Build a bridge over there", "make it night", "give me wings": there is no key for that. That is what your voice is for. A few seconds of latency is fine here. It does not feel like lag. It feels like casting a spell.

Everything else follows from that split. The character talks back right away so the wait feels like a conversation, not a loading bar. A ghost outline appears where you aimed the instant you release the button, before the model has said a word. And the model is blind on purpose: it never sees the world, only what you point at and what you say. The gap between what you asked and what she proudly delivers is where the fun lives. We tried to fix it and the game got worse.

What she can do

• Push-to-talk. Hold right click, talk, release. A ghost appears where you aimed while she thinks.
• Answer first, build second. A short spoken reply lands in under a second. The structure lands a few seconds later.
• Build, edit, undo. Towers, houses, trees, statues, bridges. Paint it red. Make it taller. Copy this over there. Delete everything. Undo.
• Come to you, fly with you. She walks over when you call her. Ask for wings and you take off.
• Switch languages mid-sentence. English or Spanish, detected per turn. Adding a language is one config entry and two prompt files.
• Never repeat herself. The words she already said are forbidden next turn. Without that, a 26B model says "amazing" forever.

The stack

All local, all open weights, one consumer GPU (tested on an RTX 3090).

The model does not emit cubes. It emits volumes with stretch labels ("tall", "flat", "wide"), which the client expands into blocks. Same token budget, objects ten times bigger. Two model calls run per turn against a single llama.cpp slot: a tiny one for the spoken acknowledgement, a bigger one for the build. The small one hides most of the latency of the big one.

Every number lives in config.js. Voice pitch, ghost size, token budgets, detail levels, language markers, key bindings. If you want her sillier, faster or bigger, change a number. You never have to read the logic.

Run it

You need Python 3, a browser (Firefox or Chrome), a GPU that fits a ~15 GB GGUF, and three local services. They are not part of this repo; anything that speaks the same contract works.

The llama-server command we run:

llama-server -m gemma-4-26B-A4B-it-UD-Q3_K_M.gguf -ngl 99 --flash-attn on \
  -c 262144 --cache-type-k q4_0 --cache-type-v q4_0 -np 1 --port 6981 --reasoning off

Then:

git clone https://github.com/franciscocarloserra/llm-sandbox && cd llm-sandbox
echo "your-local-bearer-token" > .token     # only if your STT/TTS require one
python3 server.py                          # http://localhost:7801

Override anything with env vars: SANDBOX_PORT, STT_URL, TTS_URL, LLM_URL, LOCAL_TOKEN.

Click the page, hold right click, talk.

Tests

node test/run.mjs          # 564 unit tests, no browser, no services needed
python3 test/services.py   # integration against the three live services

How it was built

By AI coding agents, several in parallel, with a human directing by voice. Two files hold the project together: AGENTS.md (the rules, the map, the traps that already cost us) and INTENT.md (the why behind every decision, measured rather than assumed). The character, the rules she follows and the output schema live in lang/*/, not in code. If you want to change who she is, edit a text file.

License

MIT. audio/Level_One_Morning.mp3 is the only bundled asset; see its own license.