An open-source runtime for realtime voice and multimodal agents — local or hosted models behind an OpenAI Realtime-compatible API.
Go
1
1,883 commits
updated Oct 10, 2026
Realtime AI that keeps talking while it works.
An open-source runtime for voice and multimodal agents. Choose local or hosted models and connect through an OpenAI Realtime-compatible API.
Get started · Demos · Documentation · Protocol · Examples
Twelve realtime scenarios, played against the real models. Interrupt the assistant and hear it pick up where it was cut off; have it count as you talk, translate as you speak, press a key while a recording still reads, and stay quiet when the words are for somebody else. The scenario gallery describes each one, and the default pipeline shows how to play them all in about a minute.
OpenRealtime brings conversation, background reasoning, and model integrations into one inspectable runtime. Use it to build voice assistants, meeting copilots, or agents that respond to changing screens.
@openai/agents-realtime SDK is
tested over WebSocket and WebRTC. Negotiated OpenRealtime extensions add
video, observations, and bounded computer use.The conference room uses the project’s twelve-scenario pipeline. It needs Linux, Go 1.25+, Git, a Chromium-based browser, Deepgram and Gemini credentials, and the local services listed in Conversation room. The Linux bound is the room's, not the project's: the room freezes a launch profile and reads it back through a hardened open implemented for Linux only. macOS runs an explicitly composed cascade instead — see Use local models.
git clone https://github.com/bojieli/OpenRealtime.git
cd OpenRealtime
go build -o openrealtime ./cmd/openrealtime
export GEMINI_API_KEY="your-key"
export DEEPGRAM_API_KEY="your-key"
./openrealtime companion
The command starts the server and browser client, then opens
http://127.0.0.1:8767. Allow microphone access, connect, and ask a question.
Keep the terminal running; press Ctrl+C to stop.
The room is the default pipeline. ./openrealtime pipelines lists the others,
such as room-flux with Deepgram Flux recognising, and
./openrealtime companion -pipeline NAME runs one; see
Choosing a pipeline.
For a first interaction, ask the assistant to explain a topic, then interrupt with a shorter follow-up. Listen for when it stops and how it responds. To hear the twelve scenarios rather than improvise, play them against the room you just started; the commands are in the default pipeline.
See the quickstart for expected output, troubleshooting, and other hosted providers. To connect your own application, start with the SDK tool-use walkthrough.
OpenRealtime connects to model servers you run; it does not bundle weights or
start those services automatically. The room's default pipeline runs the
interaction policy and speech synthesis locally and uses Deepgram for
recognition and Gemini as the voice; openrealtime profile scenario authors
the all-local variant with SenseVoice recognition and a local voice. The
local stack guide explains the endpoints and
how to make every component local.
A session shares conversation history across perception, reasoning, and action. Interaction policies decide when work should run and when output can reach the user. In the default voice arrangement, the foreground speaks and the background reasoner returns information and tool results for it to present.
The project's default pipeline is the conversation room: a cascade of a recogniser, a small policy model asked one question per transcript event, a voice model, and a synthesiser, arranged to behave the way a single duplex model would, and accepted by playing twelve scripted scenarios against the real models.
flowchart LR
Client[Browser or API client] <--> Runtime[OpenRealtime runtime]
Runtime <--> Voice[Conversational voice]
Runtime <--> Reasoner[Background reasoner]
Reasoner <--> Tools[Tools]
Policy[Interaction policies] -. when to speak or wait .-> Runtime
Choose an adapter for the voice path:
| Binding | Voice path | Use it when |
|---|---|---|
upstream | hosted Realtime endpoint | you want a first conversation without local model services |
cascade | speech recognition → language model → speech synthesis | you want to select each component |
omni | speech-to-speech model | you want native audio generation with runtime turn control |
duplex | model with concurrent audio input and output | you want the model's native interaction and turn control |
The graph runtime supports additional compositions, including alternative speech roles and silent computer use. See Architecture for the relationship between graphs and the existing binding launch paths, including work still in progress.
OpenRealtime implements a tested subset of the OpenAI Realtime API. Compatibility checks cover schema validation and tool-using sessions with the published SDK on both transports. Some events and session settings are unsupported; see the compatibility reference before migrating an application.
Video and computer use require a capable deployment and explicit extension
negotiation. The hosted upstream binding does not expose those capabilities.
Actions also require declared tools, targets, and authority; see the
protocol and safety model.
The source version is recorded in VERSION. Run
./openrealtime version to identify a built binary. The
changelog records changes, and the
release validation matrix describes the checks
behind release claims.
| Goal | Start here |
|---|---|
| Try a conversation | Quickstart · Demo gallery |
| Select models | Providers · Local setup |
| Connect an application | SDK example · Transports |
| Understand or extend the runtime | Architecture · Component API · Sidecars |
| Deploy | Deployment · Operations |
| Explore the research | Research overview · Benchmarks |
The documentation index links to the full reference, application guides, and design records.
Useful contributions include model adapters, runnable examples, interaction bug reports, and improvements to setup and error messages. See CONTRIBUTING.md for starter tasks, development checks, and the pull request workflow.
Follow the Code of Conduct in project spaces. Report vulnerabilities through the private process in SECURITY.md.
Software is licensed under the MIT License; documentation is CC BY 4.0. See LICENSES.md for original fixtures, third-party provenance, and license exceptions. Model weights and external services have their own licenses and terms.
An open-source runtime for realtime voice and multimodal agents — local or hosted models behind an OpenAI Realtime-compatible API.
Go
1
1,883 commits
updated Oct 10, 2026
Realtime AI that keeps talking while it works.
An open-source runtime for voice and multimodal agents. Choose local or hosted models and connect through an OpenAI Realtime-compatible API.
Get started · Demos · Documentation · Protocol · Examples
Twelve realtime scenarios, played against the real models. Interrupt the assistant and hear it pick up where it was cut off; have it count as you talk, translate as you speak, press a key while a recording still reads, and stay quiet when the words are for somebody else. The scenario gallery describes each one, and the default pipeline shows how to play them all in about a minute.
OpenRealtime brings conversation, background reasoning, and model integrations into one inspectable runtime. Use it to build voice assistants, meeting copilots, or agents that respond to changing screens.
@openai/agents-realtime SDK is
tested over WebSocket and WebRTC. Negotiated OpenRealtime extensions add
video, observations, and bounded computer use.The conference room uses the project’s twelve-scenario pipeline. It needs Linux, Go 1.25+, Git, a Chromium-based browser, Deepgram and Gemini credentials, and the local services listed in Conversation room. The Linux bound is the room's, not the project's: the room freezes a launch profile and reads it back through a hardened open implemented for Linux only. macOS runs an explicitly composed cascade instead — see Use local models.
git clone https://github.com/bojieli/OpenRealtime.git
cd OpenRealtime
go build -o openrealtime ./cmd/openrealtime
export GEMINI_API_KEY="your-key"
export DEEPGRAM_API_KEY="your-key"
./openrealtime companion
The command starts the server and browser client, then opens
http://127.0.0.1:8767. Allow microphone access, connect, and ask a question.
Keep the terminal running; press Ctrl+C to stop.
The room is the default pipeline. ./openrealtime pipelines lists the others,
such as room-flux with Deepgram Flux recognising, and
./openrealtime companion -pipeline NAME runs one; see
Choosing a pipeline.
For a first interaction, ask the assistant to explain a topic, then interrupt with a shorter follow-up. Listen for when it stops and how it responds. To hear the twelve scenarios rather than improvise, play them against the room you just started; the commands are in the default pipeline.
See the quickstart for expected output, troubleshooting, and other hosted providers. To connect your own application, start with the SDK tool-use walkthrough.
OpenRealtime connects to model servers you run; it does not bundle weights or
start those services automatically. The room's default pipeline runs the
interaction policy and speech synthesis locally and uses Deepgram for
recognition and Gemini as the voice; openrealtime profile scenario authors
the all-local variant with SenseVoice recognition and a local voice. The
local stack guide explains the endpoints and
how to make every component local.
A session shares conversation history across perception, reasoning, and action. Interaction policies decide when work should run and when output can reach the user. In the default voice arrangement, the foreground speaks and the background reasoner returns information and tool results for it to present.
The project's default pipeline is the conversation room: a cascade of a recogniser, a small policy model asked one question per transcript event, a voice model, and a synthesiser, arranged to behave the way a single duplex model would, and accepted by playing twelve scripted scenarios against the real models.
flowchart LR
Client[Browser or API client] <--> Runtime[OpenRealtime runtime]
Runtime <--> Voice[Conversational voice]
Runtime <--> Reasoner[Background reasoner]
Reasoner <--> Tools[Tools]
Policy[Interaction policies] -. when to speak or wait .-> Runtime
Choose an adapter for the voice path:
| Binding | Voice path | Use it when |
|---|---|---|
upstream | hosted Realtime endpoint | you want a first conversation without local model services |
cascade | speech recognition → language model → speech synthesis | you want to select each component |
omni | speech-to-speech model | you want native audio generation with runtime turn control |
duplex | model with concurrent audio input and output | you want the model's native interaction and turn control |
The graph runtime supports additional compositions, including alternative speech roles and silent computer use. See Architecture for the relationship between graphs and the existing binding launch paths, including work still in progress.
OpenRealtime implements a tested subset of the OpenAI Realtime API. Compatibility checks cover schema validation and tool-using sessions with the published SDK on both transports. Some events and session settings are unsupported; see the compatibility reference before migrating an application.
Video and computer use require a capable deployment and explicit extension
negotiation. The hosted upstream binding does not expose those capabilities.
Actions also require declared tools, targets, and authority; see the
protocol and safety model.
The source version is recorded in VERSION. Run
./openrealtime version to identify a built binary. The
changelog records changes, and the
release validation matrix describes the checks
behind release claims.
| Goal | Start here |
|---|---|
| Try a conversation | Quickstart · Demo gallery |
| Select models | Providers · Local setup |
| Connect an application | SDK example · Transports |
| Understand or extend the runtime | Architecture · Component API · Sidecars |
| Deploy | Deployment · Operations |
| Explore the research | Research overview · Benchmarks |
The documentation index links to the full reference, application guides, and design records.
Useful contributions include model adapters, runnable examples, interaction bug reports, and improvements to setup and error messages. See CONTRIBUTING.md for starter tasks, development checks, and the pull request workflow.
Follow the Code of Conduct in project spaces. Report vulnerabilities through the private process in SECURITY.md.
Software is licensed under the MIT License; documentation is CC BY 4.0. See LICENSES.md for original fixtures, third-party provenance, and license exceptions. Model weights and external services have their own licenses and terms.