SignWave

Two-way, real-time communication between a Deaf patient who signs American Sign Language (ASL) and a speaking clinician in an emergency room. Everything runs live on one MacBook. Nothing is pre-recorded, and the AI output is never faked.

• Doctor → patient: the doctor speaks. The patient reads live captions and watches a 3D cartoon cougar mascot sign the words, word by word, posed directly from Deaf signers' recorded motion. The doctor can correct any caption.
• Patient → doctor: the patient signs. The app translates the signing into English candidates, and the patient confirms, corrects or rejects them before anything is shown to the doctor as "patient said".

Research prototype. SignWave is not a certified medical interpreter. Do not use it for clinical decisions. Use a qualified ASL interpreter for care.

What it can and cannot do

Natural signing is switched off for now. The patient view offers one sign at a time only. The natural-signing code is commented out in medbridge/server.py and web/src/components/PatientView.tsx; the natural-signing parts of this README describe it for when it is switched back on.

It cannot:

• translate like a human interpreter;
• understand signs outside the training data (for the precise mode, outside the vocabulary);
• reliably read fingerspelling;
• guarantee any translation.

That is why every patient message is confirmed by the patient, and why unconfirmed model output is always labelled as such on the doctor's screen.

Why these two models: no open model for continuous ASL → English translation performs better (checked October 2026).

• Google's SignGemma has not been released.
• Google's SL2T runs only on Pixel 11 and has no API.
• General video LLMs score about 1 BLEU on ASL: Gemini-2.5-Pro 1.4, GPT-5 0.1 on How2Sign.
• SHuBERT has released no translation checkpoint.

Uni-Sign is the strongest open continuous model. ASL Citizen is the strongest open isolated-sign model and is trained on webcam video from 52 Deaf or hard-of-hearing signers.

Measured results

All numbers come from running our live pipeline code on held-out data (scripts/eval_*.py).

Notes:

• A larger pose model (rtmlib performance) scored lower (BLEU-4 11.4) and runs at about 6 fps, so lightweight is used.
• The ASR numbers use synthesized speech. Run python scripts/eval_asr.py --live with a real speaker to measure the real case.
• Optional cloud speech-to-text (ASR_ENGINE=elevenlabs, ElevenLabs Scribe v2 for the final captions): on the same 80 clips on an M2 Pro, WER 0.9%, medical-term recall 100%, p50 514 ms / p95 769 ms, against Whisper's 5.6%, 94% and 616 / 659 ms on that machine. Nearly all of the gap is one voice (macOS "Aman": Whisper 18.8%, Scribe 0%); on the other three both are near 1%. Reproduce with python scripts/eval_asr.py --engine elevenlabs.
• The live signing accuracy depends on the signer. Measure it with the protocol in Testing.

Architecture

                         Mac A (patient laptop, runs everything)
 Logitech cam ─► vision.py camera thread (unmirrored frames, memory only)
   ├─ natural ─► rtmlib whole-body pose ─► sentence segmenter ─► Uni-Sign (mT5) on MPS ─► top-3 sentences
   ├─ precise ─► MediaPipe Holistic ─► sign segmenter ─► ST-GCN ─► top-5 signs ─► translate.py
   └─► landmarks (WebSocket, patient view only) + MJPEG preview
 Mic ─► asr.py: Silero VAD ─► utterance segmenter ─► mlx-whisper ─► interim / final captions
   └─► final caption ─► text2sign.py: word → ASL Citizen sign (Deaf signer motion, local cache)
                         or sign.mt (missing words only) ─► joined 3D motion ─► patient's signing figure
 hub.py: session state, role-checked actions ─► WebSocket snapshots ─► /patient and /doctor

• Device capture happens in Python, so the doctor's laptop needs no camera or mic permission. It only opens a web page.
• Only the active mode's pose model runs.
• If GLOSS_TRANSLATOR=llm, Claude rephrases precise-mode sign labels. Natural-signing output is never rewritten by an LLM.

Setup (macOS, Apple Silicon)

Requirements: Python 3.11, Node 20+, Homebrew ffmpeg (only for fetch_eval_clips.py).

python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python scripts/fetch_models.py           # ~3 GB: Uni-Sign, ASL Citizen weights + gloss list, whisper
python scripts/build_sign_lexicon.py     # one-time ~10 min: signing motion for common doctor words
(cd web && npm install && npm run build)
cp .env.example .env                      # then set CAMERA_INDEX / MIC_NAME
python scripts/check_devices.py           # finds the Logitech camera + mic, checks permissions

Run

python -m medbridge.server

• Models load in about 20 s. The UI shows "Loading…" until they are ready.
• The camera stream and the patient controls are served only to Mac A itself. Other machines on the network can open only the doctor view. Set PATIENT_LAN=1 to allow the patient view from another machine.
• Frontend development with hot reload: run cd web && npm run dev, then open http://localhost:5173/patient. The dev server proxies /ws, /video.mjpg and /api to port 8000.

Live demo procedure

1. Setup. Put the Logitech camera at chest height, with even lighting from the front and a plain background. The patient sits about 1 m away, waist up in frame, hands resting in the lap.
2. Doctor: "Hi, I'm Doctor Lee. What brought you in today?" Captions appear live on the patient screen.
3. Patient (natural signing): signs a short sentence, then rests their hands.
   • Three English candidates appear. The patient taps the right one and presses Confirm & send.
   • The doctor sees it with "✓ Confirmed by patient".
4. Doctor: "On a scale of one to ten, how bad is the pain?"
   • The patient switches to One sign at a time, signs a number, and confirms.
   • FIVE and TEN are not in the vocabulary. Use 1–4 or 6–9.
5. Doctor: "Are you allergic to penicillin?"
   • If a drug name is mis-transcribed, the doctor clicks the caption and fixes it.
   • The patient sees "Corrected by doctor" with the changed word highlighted.
6. Honesty moment. Sign something unusual.
   • Show the "Low confidence" flag or the "Not sure — pick" chip, then the Retry and Signs I can recognize panels.
7. Finish. Show the doctor's System panel (fps and latency), then the results table above.

Tips:

• Rehearse the precise-mode signs with the ASL-LEX / Signing Savvy videos for the exact citation forms.
• Pause briefly before and after signing.

Testing and measurement

Step-by-step test examples for one Mac (signs, sentences, doctor lines, what to record) are in docs/testing.md.

pytest                                   # 69 tests: parity with upstream preprocessing, segmenters, hub, server, ASR
(cd web && npx vitest run)               # frontend helpers
python scripts/fetch_eval_clips.py --n 60 && python scripts/eval_unisign.py --pose-mode lightweight
python scripts/eval_islr.py --n-random 150 --per-gloss 3
python scripts/eval_phrase_bank.py       # natural signing + ER phrase bank
python scripts/eval_asr.py               # add --live for a real speaker
METRICS_LOG=1 python -m medbridge.server # logs timings/scores/labels (no text/audio/video) to data/metrics.jsonl

Live signing protocol (Phase 6):

• Precise mode: sign each ER sign 5 times and note top-1 and top-5 per sign.
• Natural mode: sign the 20 patient sentences in data/er_phrases.json. Judge whether the meaning is right in the top-1 and in the top-3.
• False recognition: spend 2 minutes not signing (talking, scratching your face, adjusting glasses). Count the outputs that are not flagged.
• Successful exchanges: run 10 scripted question-and-answer exchanges. Count those confirmed with the correct meaning and without typing.

Troubleshooting

Privacy and safety

• Audio and video are processed in memory and never written to disk. No recordings are made.
• Speech recognition runs locally, unless ASR_ENGINE=elevenlabs is set: then the audio of each thing the doctor says is sent to ElevenLabs for the final caption.
• Network calls:
  • optional Claude rephrasing of precise-mode sign labels;
  • for the doctor-to-patient signing, any word that is missing from the local sign dictionary is sent to the public sign.mt service. Set SIGN_MT=0 to keep everything on the Mac; missing words then show as text chips.
  • if ELEVENLABS_API_KEY is set, the text of each confirmed patient message is sent to ElevenLabs, which returns the audio the doctor view plays. Leave the key empty to keep this off.
• The conversation lives in memory until End session.
• There is no diagnosis or clinical decision support.

Project layout

medbridge/   server.py hub.py vision.py segmenter.py asr.py translate.py metrics.py
             unisign/ (trimmed Uni-Sign pose model)  unisign_runner.py
             islr.py  stgcn/ (vendored ASL Citizen ST-GCN, MIT)
web/         React + Vite + Tailwind + Motion: patient (dark) and clinician (light) views
             src/theme.css holds every colour, as five themes (classic is the default; cougar, ocean, midnight, sage);
             add ?theme=NAME to the address to try one, or edit the file and `npm run build`
scripts/     check_devices, fetch_models, fetch_eval_clips, eval_unisign, eval_islr, eval_asr
data/        medical_vocab.json, er_phrases.json (everything else here is downloaded and gitignored)
tests/       pytest suite

Attribution and licenses

• Uni-Sign: Li et al., Uni-Sign: Toward Unified Sign Language Understanding at Scale, ICLR 2025. The model and preprocessing code in medbridge/unisign/ are adapted from https://github.com/ZechengLi19/Uni-Sign. The upstream repository has no license file, so treat this use as research and evaluation only.
• ASL Citizen: Desai et al., NeurIPS 2023 Datasets & Benchmarks. The code and weights are MIT (medbridge/stgcn/LICENSE). The dataset (only its split CSVs and a few test videos are fetched) is under the Microsoft Research License Terms in use.txt.
• OpenASL: Shi et al., 2022. Test clips are fetched from YouTube only for offline evaluation and are not redistributed.
• Whisper: OpenAI, run through mlx-whisper.
• Silero VAD: MIT.
• rtmlib / RTMPose: Apache-2.0.
• MediaPipe: Apache-2.0.