Textured

Sounds of words.

A musical toy from Fluxus, an AI product studio. An instrument you play with words.

Play Textured → · Set up your own copy

A sound description can be a preset you can read and share. Textured explores how far a few typed judgments can take that idea: TypeSafe interprets your words, and code maps those judgments onto a synthesiser running in your browser. Turn the knobs to make it yours, or share the words for someone else to interpret. Each interpretation can be a little different.

Set up your own copy

You need Git, Node.js 24 (recommended; minimum 22.13), and a TypeSafe API key to turn words into sounds. Jev is the model; the key comes from TypeSafe. Playback and manual controls work without a key. Running locally needs no Cloudflare account or C++ toolchain.

1. Sign in to the TypeSafe console and create an API key, following their quickstart. If your account does not have access yet, check TypeSafe's access options.
2. Use the coding-agent prompt below, or follow the manual commands. Keep the key in your local .env.local file; it is ignored by Git. Do not paste it into a chat, commit it, or put it in browser code.
3. Open the local player, type a nervous elevator, and click Make sound. This calls TypeSafe using your account; its API usage is billed separately from your coding agent. Play on the initial patch needs no API call.

Give this to your coding agent

Paste this into your coding agent:

Set up https://github.com/djangobeatty/textured locally using its README, help me add my TypeSafe API key privately, and open the player.

For changes to the Jev questions or mappings, the optional official TypeSafe coding-agent skill is useful.

Run locally by hand

git clone https://github.com/djangobeatty/textured.git
cd textured
npm ci
cp .env.example .env.local

Open .env.local in your editor and set TYPESAFE_API_KEY to your own key (or leave it blank for manual playback). Keep TURNSTILE_REQUIRED=false locally. If you already have a checkout or .env.local, reuse it instead of overwriting it.

npm run build
# First run only: initialize local storage
npm run db:setup
npm run dev

Stop the server with Ctrl+C; restart it with npm run dev. Restart after changing .env.local. To host your own public player, follow the separate Cloudflare deployment guide.

Check it is working

At the local URL, the page should show today's description allowance. After adding a key, Make sound should change the controls and start the sound. If it reports that TypeSafe is not connected, check TYPESAFE_API_KEY in .env.local and restart the server. If the allowance cannot be checked, run npm run db:setup from the repository root. A rejected key needs checking in the TypeSafe console.

Playing the instrument

Click Make sound to interpret your words, or Play to hear the initial patch. Edit the description while playing to reshape the sound after a short pause in typing. Play/Stop, voices and blend, playing style, volume, and all knobs and sliders work independently of inference. The initial patch is explicitly hand-tuned; it works without a key. The first play loads the audio engines.

Try An underwater cathedral, slowly waking. Keep Movement low and turn up Slow shifts for a slowly evolving drone. Change Texture from 0 to 100 to move through the engine's timbre and morph parameters and add saturation. Increase Movement for wider, less repetitive melodies, irregular gates and faster timbral modulation.

Scale sets the notes the instrument can use: Marbles' Major, Minor, Pentatonic, Pelog, Bhairav and Shri presets, plus Chromatic for all twelve semitones. TypeSafe selects a scale from your description; the selector overrides it immediately. The tonic is C. Pelog, Bhairav and Shri preserve the firmware's microtonal tuning. These are the module's preset interpretations, not a complete implementation of a musical tradition.

Movement varies the phrase and range within the scale. Marbles' pitch voltages retain their original 1V/octave relationship, and a final quantizer prevents out-of-scale target notes. Floating chords choose compatible chord shapes; microtonal scales fall back to octave layers where no fixed Plaits chord fits. Portamento and subtle drift still move the sounding pitch around these targets.

Drone notes use a light portamento (roughly 75ms to travel 95% of an interval). Slow shifts adds subtle pitch drift (up to ±18 cents, scaled by Tension) alongside the evolving timbre and space.

Patch values ease from the current sound to their new settings over 750ms. Sound-family and playing-style changes crossfade between two preallocated voice pairs; only the active pair renders once the fade finishes. Rapid family changes finish the current fade and then move to the latest requested family. Effect parameters settle over roughly the same time. Scale selection retunes note targets immediately, preserving the short note glide.

How it works

TypeSafe interprets a short description; the original Plaits and Marbles DSP, compiled to WebAssembly, turns the resulting typed judgments into sound inside a Web Audio worklet.

One request asks eleven independent questions over the same description:

• Seven Score questions measure energy, brightness, tension, acoustic space, movement, texture, and slow evolution. Each has five descriptive levels; its 0–4 score maps to a 0–1 control.
• A Choice question returns probabilities over eight sound families. The two highest probabilities are normalized and used to blend two independently rendered Plaits voices. The UI shows all eight original probabilities in a labelled bar chart. Voice A/B selectors and a blend fader edit the actual two-voice mix, with solo positions at each end; a new interpretation resets the voices and balance. The six Character knobs plus Slow shifts are the seven continuous model controls. This is an artistic mixture, not an uncertainty claim about the sound.
• A Choice question selects major, minor, suspended, or chromatic harmony.
• A Choice question selects the melodic scale. The harmonic preference above is used only when its chord tones fit the scale.
• A Choice question selects sustained/drone, rhythmic/pulse, or scattered/pluck articulation.

lib/semantic.ts contains the questions, typed patch and validation. public/audio/semantic-processor.js runs two Plaits voices and a Marbles generator on the audio thread at the firmware's 48 kHz, resampling when necessary. Marbles generates the gates and pitch voltages; there is no fixed melody. Slow shifts overlap 23–71 second cycles for timbre, harmonics, stereo balance, filtering and reverb. The native Web Audio effects and transport live in lib/synth.ts. app/api/interpret/route.ts keeps the API key on the server.

The page also registers two WebMCP tools, read_synth_patch and interpret_sound_description, in hooks/use-instrument-tools.ts. WebMCP is a draft browser API that lets a page offer functions to an AI assistant built into the visitor's browser. An assistant could read the current controls or send a description through the same interpretation path a visitor uses, with that visitor's cookie and allowance. Nothing is exposed over the network and there is no MCP server. No shipping browser supports the API yet, so the registration is inert until one does.

The model does not generate audio or hear the output. The musical mapping is authored in code. Model judgments are subjective and can be wrong; the knobs always remain yours. API latency is measured on the server, including retries. Cloudflare's native Cache API retains a successful interpretation for five minutes. Repeated descriptions within that window can reuse it and are labelled cached; the reported latency belongs to the original inference. The key includes the model, questions and description. Outward API responses remain private/no-store so their per-browser allowance is never shared. Sound playback never waits for network calls after starting.

Embedding

A single script tag mounts the full player and controls directly in another page. See the embedding guide for the snippet, optional prefilled sound, Cloudflare hosting requirements and session/security behaviour.

Sharing

Share this sound copies only the description, for example ?sound=dolphins+swimming. In an embed, the link uses the containing page's current domain and path, so moving the host page to a custom domain needs no embed-code change. Opening the link prefills the words; the visitor asks TypeSafe to interpret them again. Knob positions, audio and the playhead are not included. Previously issued patch links still open, but new links contain no patch/settings fragment.

Drag any of the six rotary knobs up/right to increase or down/left to decrease. Hold Shift for fine adjustment, use arrow keys for steps and Page Up/Down for larger changes, or double-click to restore that knob's initial setting. Knobs, Slow shifts and Volume work without inference.

Development

npm run check

This runs lint (including zero warnings), TypeScript, every test suite and the production build. It needs no API keys and makes no paid calls. GitHub Actions runs the same command for pushes to main and pull requests.

After the first build and npm run db:setup, use npm run dev for hot reload. npm start serves the last production build locally; rebuild to see source changes. Neither command deploys anything. For your own Worker, follow the Cloudflare deployment guide.

The code is split by responsibility:

Targeted checks are available as npm run test:instrument, test:semantic, test:share, test:embed, test:quota and test:dsp. The React tests cover hydration of shared descriptions, playback, stale-response cancellation, manual edits, legacy patch links, trusted parent messages and clipboard fallback.

Uses React, TypeScript, Vinext, and the native Web Audio API. Production server output targets Cloudflare Workers. Set TYPESAFE_API_KEY as a runtime secret on your host. Never prefix it with a public/browser environment prefix. Cloudflare serves and caches static assets; successful inference results use its native Cache API with a five-minute TTL, without an in-memory fallback cache. Daily allowances are persisted in D1.

Daily allowance

The server permits 50 submitted descriptions per UTC day by default, configured with TYPESAFE_DAILY_LIMIT (0 pauses interpretation). Every verified submission reserves one slot before checking Cloudflare's result cache or calling TypeSafe. Repeated descriptions (including cache hits) and failed upstream attempts count; the existing single upstream retry belongs to the same description slot. This is a description allowance, not a precise cap on provider billing. Playback and manual controls do not use it; interpreting a shared description does.

Visitors receive a random 256-bit token; the server stores only its hash, and tokens expire after 30 days. Standalone players keep it in a Secure, HttpOnly, SameSite cookie. Embedded players also keep their token inside the player's own browser storage and send it directly to the API, so interpretation works when Safari blocks third-party cookies. If browser storage is also unavailable, it stays in memory for that page visit. See embedded sessions for details. Reloads reuse a persisted token; there is no periodic token refresh that resets the counter. Clearing storage or obtaining another session can yield another allowance. Only a valid server-issued token selects an allowance; caller-chosen user IDs do not. This does not identify a person or impose an overall paid-call cap. The standalone Cloudflare deployment requires a fresh, server-verified Turnstile token for each description; session-issuance throttling and an overall spending ceiling remain separate controls.

The D1 conditional insert/update reserves each slot atomically, including concurrent requests across Workers. Each UTC date has its own counter. Exhausted requests return HTTP 429 with the remaining allowance, reset time and Retry-After; unavailable quota storage fails closed before any TypeSafe call. The UI shows the allowance and leaves the synth controls usable at the limit.

npm run test:quota exercises real local D1 storage, concurrent requests, midnight rollover, separate browsers, browser tokens, and the API route with stubbed TypeSafe and Turnstile services, including invalid/replayed tokens and hostname/action checks. It makes no paid API calls. Schema changes belong in db/schema.ts; run npm run db:generate and apply new local migrations in order. Apply them to production with npx wrangler d1 migrations apply DB --remote --config cloudflare/wrangler.jsonc. For hot-reloading development, use npm run dev after initializing local D1.

The DSP test renders the real shipped worklet and WebAssembly in a Node harness. It checks every sound family, finite/bounded output, audible differences from Texture and Slow shifts, increased melodic variation from Movement, 44.1 kHz resampling, stop/resume, scale membership across all three gestures, compatible chord tones, live scale changes and microtonal tuning. It does not replace an actual listening test on target devices.

The precompiled binaries are checked in, so normal app development needs no C++ toolchain. To rebuild them, install Emscripten 4.0.20, activate its environment, then run python3 scripts/build-dsp.py. The script fetches pinned upstream revisions. DSP_EURORACK and DSP_STMLIB can point to local checkouts at those exact commits. The C wrappers in dsp/ are adapted from NoodleRack's build chain.

Contributions: improve the sound mappings, add clearer rubric examples, or extend the engine. Keep numeric mappings explicit and keep network calls outside audio processing.

License

MIT. Plaits, Marbles and stmlib DSP are by Émilie Gillet; the source wrappers are by stets. Full attribution and MIT terms ship in public/audio/NOTICE.txt and public/audio/LICENSE.txt. Mutable Instruments is not affiliated with this project. Dependencies retain their respective licenses.