Ambient Scribe

Real-time medical transcription system that captures clinical conversations, performs speaker diarization and automatic speech recognition, and infers DOCTOR/PATIENT roles -- delivering labeled transcripts to the browser in real time. After Stop, a second ASR pass corrects the transcript and an off-GPU agent drafts a citation-linked SOAP note for clinician review.

Consultation Flow

While recording

1. Transcribe and clean speech: Multitalker Parakeet creates timestamped words, then medical_lexicon.txt corrects known ASR variants before they reach the live transcript.
2. Identify speakers: Sortformer labels detected speakers with IDs such as spk_0 and spk_1.
3. Assign roles: The role agent maps each speaker ID to DOCTOR or PATIENT.

After recording stops

4. Correct the transcript: By default, Parakeet Unified automatically re-transcribes the full recording when it is no longer than 15 minutes.
5. Refine role labels: Corrected words keep their live labels, changing only when text provides a clear role cue. Full speaker diarization does not run again.
6. Generate the summary: Clicking Generate summary uses the corrected transcript. If correction is unavailable and the live transcript remains verified, the app uses that instead.

Architecture

+-----------+  GET /scribe + helper proxies   +--------------+
|           | ------------------------------> |  Symfony App |
|           |                                 |  (PHP 8.3+)  |
|           |                                 +------+-------+
|           |                                        | proxied HTTP
|  Browser  |  WebSocket (16 kHz PCM audio)   +------v-------------------+
|           | ------------------------------> |  FastAPI NeMo agent      |
|           |                                 |  - Diarization (GPU)     |
|           |                                 |  - ASR Parakeet (GPU)    |
|           |                                 |  - Post-stop 2nd-pass ASR|
|           |                                 |  - Role inference        |
|           |                                 |    DOCTOR / PATIENT      |
|           |                                 |    (Strands, off-GPU)    |
|           |                                 |  - SOAP summary (off-GPU)|
|           |                                 +------+-------------------+
|           |                                        | publish raw/roles/summary
|           |  Mercure SSE (transcript events) +-----v--------+
|           | <------------------------------- |  Mercure Hub |
+-----------+                                  +--------------+

Prerequisites

• Docker and Docker Compose
• NVIDIA GPU with CUDA support and 16 GB VRAM - loading both speech models peaks at 11.3 GB, and buffered audio pushes it higher across a long consultation
• NVIDIA Container Toolkit installed on the host
• 16GB+ system RAM recommended
• AWS credentials configured for Bedrock access

Quick Start

cp .env.example .env
docker compose up --build

The application will be available at http://localhost:48082.

The copied .env selects AWS Bedrock for role inference and summaries (ROLE_AGENT_MODEL_PROVIDER=bedrock), so Bedrock credentials must be present. For a fully local CPU stack, set ROLE_AGENT_MODEL_PROVIDER=ollama and start with the ollama Compose profile. ./scripts/start-dev.sh is the guided local-dev entry point: it enables the profile automatically and runs health checks.

Tech Stack

Development

composer test          # PHPUnit
composer analyse       # PHPStan level 10
composer cs:check      # PHP-CS-Fixer dry-run
composer preflight     # All quality checks in sequence
strands_agents/.venv/bin/pytest tests/python/ -q   # Python agent tests

Documentation

• README_summary.md - plain-English overview and model inventory
• README_HOW_IT_WORKS.md - detailed system walkthrough
• README_STACK.md - full model, service, and dependency inventory
• README_CLINICAL_INTELLIGENCE.md - medical term normalisation and summary grounding
• README_FIXTURES.md - demo/replay audio fixture setup
• CHANGELOG.md - feature-by-feature history

Author

Built by Matthew Hansen.