Speech to Decisions with Granite

Talk into the microphone and watch a model size up what you're saying, while you're still saying it.

One model does everything. Granite Switch 4.1 8B is Granite 4.1 8B with speech recognition and adapters built in, all served from one Hugging Face Inference Endpoint:

• Granite Switch: ASR Encoder turns the audio into text. It's a cascade inside the endpoint: a small Granite Speech model (granite-speech-5.0-470m-turboctc, a 470M English CTC encoder) transcribes each clip, and the transcript goes into the prompt as ordinary text.
• Granite Switch: Base Model says what the speaker is talking about, in 1 to 5 words.
• Granite Switch: Guardian Adapter (guardian-core) screens the words for harm, profanity and violence.
• Granite Switch: Decisions Adapter (decision) answers the yes/no questions.

Adapters are picked per request by name. Both return a noul: the probability that the answer is yes.

One KV cache for every judgment

Every request that judges a snippet of speech reuses the same KV cache. Each time the transcript changes, the Space sends 14 requests at once for the default questions: 3 safety checks, 10 yes/no questions and the label. Every one opens with the same user turn holding the words, and only what comes after it differs. So the endpoint prefills the words once, and every request reuses that KV cache.

The adapters don't break the sharing. Both are aLoRAs (activated LoRAs): the adapter switches on at an invocation marker after the words, so the words' turn runs on the plain base model in every request, the label's included. Nothing is computed twice, and the whole board comes back in about 0.2 s: 18 decisions and a label, from one model.

You can see the reuse by timing it. With a long words turn the endpoint had never seen (about 6,000 words, so that prefilling it dominates), the first decision took 0.39 to 1.1 s. A second decision with a different question on the same words took 0.15 to 0.17 s, a guardian-core check 0.23 s and the label 0.21 s; a fresh set of words went back to about 0.4 s. The cached words are shared across questions, across the two adapters and with the base model. Speech recognition is the one step outside it: its prompt holds the audio clip, and the words it returns are what everything else then shares.

This is a remake of the Speech to Decisions demo, which ran on three models through OpenRouter: Parakeet (speech recognition), Jev (the safety check and the yes/no questions) and gpt-oss-120b (the label). Here Granite Switch does all four jobs.

What's on the page

The left column has the microphone and, under it, the live transcript. Tap the microphone to start; the dots around it swell with your voice, and a dotted meter and a timer run while it listens. The right column is the board, which updates about twice a second:

• The speaker is talking about. A topic in 1 to 5 words from the base model, such as "printer jam" or "trip to Portugal".
• Safety check. guardian-core answers one question per category: is this harmful, is there profanity, is there violence. The box reads "Nothing harmful" while all three nouls are at 0.5 or below, and turns red and reads "Flagged" when any is above it. The three bars show each noul.
• Yes/No questions. A bar per question, each a noul from the decision adapter. A bar turns blue above 0.5.

Or play an example. Under the microphone, Pizza Order plays a 24-second recording of a phone order for pizza instead. The clip is static/examples/pizza-order.mp3, taken from the demo video's audio. It goes through the same path as the microphone: decoded in the browser, sent in half-second chunks at the speed it was spoken, and played through the speakers so you hear what's being judged. It stops by itself at the end, or press Stop. No microphone is needed. The clip was boosted about 12 dB from the video's audio: at its original level the room tone between words fell below the speech threshold, and the board blanked between phrases.

The yes/no questions are editable until you start recording. Each row's question sits in a text field: change it, remove it with the × at the end of its row, or type in the dashed empty row at the end to add one. There are 10 to start with, and you can add your own up to 50 in all. A full board of 50 comes back about as fast as a short one: 53 decisions took 0.25 to 0.28 s, against 0.24 to 0.29 s for 18. While the microphone is on the questions are locked and the rows are plain text, sorted so the likeliest answers sit at the top; they unlock, back in their own order, when you stop. If you edit a question after a recording, the new list is answered about the words already on the page.

Start over in the header clears the transcript and the board and keeps your questions. The i button opens a panel that explains the page.

Silence blanks the board. After 0.5 s with no speech while recording, every judgment is wiped: the label, the safety check and the yes/no bars all show a dash, and a "Waiting for speech" tag appears. No judgment is shown again until you start talking: answers still coming back from before the silence are dropped. When you talk again the board fills back in, and those judgments are about the words said after the silence only, so in practice the board judges the current stretch of speech. The transcript keeps everything.

A phrase short enough to end before the board has judged it ("hello there") would otherwise never be judged: the silence falls while it's still being transcribed. So when a silence falls on words whose judgments never reached the board in full, those words are judged once they're transcribed, and the answers show through the quiet for 3 s, or until you start talking again.

The silence is timed from the last 100 ms frame with speech in it, not from the last half-second chunk, so the board blanks 0.5 s after you stop, not up to a second later.

How Granite is called

Every call is a chat completion to the endpoint's OpenAI-compatible API. Both adapters are aLoRAs, switched on by a control token in the last user turn, so the adapter is active only from there on. Everything before it runs on the base model. An adapter is chosen with chat_template_kwargs: {"adapter_name": ...}, and the chat template puts its control token at the adapter's marker: <guardian> for guardian-core, ### DECISION for decision. Both are called as the model card shows.

Each time the words change, one pass sends 3 + N requests at once, one per safety category and one per question, plus the label beside them, and the endpoint batches them. All of them open with the same user turn holding the words (see One KV cache for every judgment).

The label runs on its own thread, beside the decisions, so it never holds them up. Up to 3 label requests run at once, the newest answer wins, and a request is given up on after 5 s.

How live transcription works

The endpoint has no transcription route. Audio goes in as an input_audio part of an ordinary chat message; before the model runs, vLLM hands the clip to the checkpoint's speech recognition model and swaps the clip's <|audio|> marker in the prompt for the transcript's tokens. So to get the transcript on its own, the Space sends a user turn holding only the clip, asks for one token, and reads the prompt back from prompt_logprobs, which spell out every prompt token the model saw. That costs one generated token, not a repeat of the transcript. (vLLM's prompt_text field would be simpler, but this endpoint leaves it empty.)

Each request takes a whole clip and returns its text; there's no streaming mode. The Space gets a live transcript out of it like this:

1. Chunks. The page captures the microphone at 16 kHz and posts it to the Space every half second as 16-bit PCM. Each chunk is appended to the session's open segment. A browser that can't capture at 16 kHz sends its own rate, and the Space resamples.
2. Re-transcribe. Once half a second of new audio has arrived, the whole open segment is transcribed again. Its text replaces the segment's live text, shown in grey, which can still change as more audio gives the recognizer more to go on.
3. Commit at a pause. When the speaker has been quiet for 0.7 s and the segment is at least 4 s long, its text is committed to the transcript and a new segment starts. A speaker who doesn't pause is cut after 15 s, at the quietest moment of the segment's last 4 s.
4. Skip the quiet. Audio with no 100 ms frame above the loudness threshold isn't sent at all, so the recognizer isn't asked to find words in room noise (given pure silence, it hears "the").

Transcription and the decisions run on a background thread per session, so the microphone stream never waits on a model.

Speed

Measured from a laptop, replaying an 11-second synthesized clip through the app's handlers against a warm endpoint:

The status line under the transcript shows the latest timings. Nothing is billed per request: the endpoint bills by the hour while it's running.

How it's built

The page is hand-built, with no front-end framework, and a small FastAPI server sits behind it. The keys stay on the server.

The API is five routes. Every POST returns the session's whole state as JSON, and the page redraws from it:

The design follows IBM's Granite demo pages, in particular Granite Speech Streaming WebGPU (Apache-2.0): the Carbon design tokens, IBM Plex type, the app card, the record panel with its cluster of dots, the dark info panel, and the tags and pills are adapted from that Space's stylesheet.

To run it locally:

pip install -r requirements.txt
HF_TOKEN=... GRANITE_ENDPOINT_URL=https://<endpoint>/v1 \
  PORT=7860 python app.py   # http://localhost:7860
pytest tests -q             # offline tests

Setup

1. Create a Docker Space (free CPU hardware is enough) and push this folder to it, or run scripts/create_speech_to_decisions_granite_space.py from the repo root.
2. Settings → Variables and secrets, add:
   • GRANITE_ENDPOINT_URL: the Granite Switch 4.1 8B endpoint URL, ending in /v1. The script copies it from HF_ENDPOINT_URL_SWITCH_8B in .env.
   • HF_TOKEN: a token that can call the endpoint. Without either, the page loads and says which is missing.
   • GRANITE_MODEL_ID (optional): the endpoint's served model name. Read from /models when it isn't set.
   • VOICE_RMS (optional): the loudness that counts as speech, as a fraction of full scale. Defaults to 0.01. Lower it for a quiet microphone.
   • QUIET_AFTER_S (optional): how many seconds of silence blank the board. Defaults to 0.5.
   • MIN_NEW_AUDIO_S (optional): how much new audio triggers the next transcription. Defaults to 0.5.

The browser asks for microphone access, which needs HTTPS. A Space has it; so does localhost.

Caveats

• The endpoint scales to zero. The first request after it's been idle waits 1 to 5 minutes while it starts; the page shows "starting up" until then. Opening the page sends a wake-up request, so it's usually up by the time anyone talks.
• The transcript is lowercase with no punctuation, and English only: that's how the checkpoint's default CTC recognizer writes ("i promise i will file the ticket today"). The model reads it that way too. A checkpoint composed with a generative recognizer such as Whisper (asr_model_id in its config.json) restores case and punctuation and handles other languages.
• A pause of 0.5 s is short. The board blanks at most sentence breaks, and after one the judgments start over from the new words. Words said just before a pause are still counted with the speech before it: the silence is marked in the audio, so they're committed with that speech, not carried into what's said after it.
• The board stays blank after you stop if you paused before pressing Stop, and editing a question then has no words to answer about. Start talking again to fill it.
• The grey text is provisional. Words near the end of the open segment often change on the next pass, and the answers change with them.
• The model sees only the most recent words, with no earlier context, so a pronoun or a half-finished sentence can swing an answer, and a name said a while ago no longer counts as mentioned.
• The safety check is three guardian-core calls with a 0.5 cut-off that hasn't been tuned. It isn't a moderation system. The categories overlap: a frustrated "i swear i am going to smash this damn thing" (about a printer) scored 0.75 on harm and 0.88 on violence as well as 0.97 on profanity.
• The label is written fresh each time, so its wording can change from one second to the next while the speaker's meaning hasn't.
• A segment cut at 15 s without a pause can split a word across two segments.
• The loudness threshold is fixed. In a noisy room, background sound can count as speech and get transcribed. It also resets the 0.5-second silence count, so a noisy room may never blank the board.
• Visitors share the Space's endpoint, so a public Space keeps the owner's endpoint awake and billing.