> ## Documentation Index > Fetch the complete documentation index at: https://docs.aihubmix.com/llms.txt > Use this file to discover all available pages before exploring further. # Realtime Conversation > Establish a persistent WebSocket connection for low-latency, two-way voice and text interaction with a conversational speech model ## Introduction Realtime conversation establishes a persistent connection over WebSocket (a protocol that keeps a long-lived, bidirectional connection between client and server), streams your audio or text input to the conversational model in realtime, and the model pushes back text and speech replies incrementally. It suits voice assistants, realtime Q\&A, spoken practice, and other scenarios that require back-and-forth interaction. Like [realtime transcription](/en/api/realtime-transcription), it runs over WebSocket, but the two serve different purposes: | Dimension | Realtime transcription | Realtime conversation (this page) | | ------------------------------ | --------------------------------------- | ---------------------------------------------------------------- | | Goal | Turn speech into text | Hold a multi-turn conversation where the model generates replies | | Direction | One-way: audio in, text out | Two-way: audio or text in, text and speech out | | Connection parameters | `?intent=transcription&model=...` | `?model=...` only (no `intent`) | | Voice activity detection (VAD) | Not supported, must be `null` | Supported, automatic turn detection available | | Typical use cases | Meeting captions, live-stream dictation | Voice assistants, realtime spoken interaction | **Available model:** * **gpt-realtime-2.1**: conversational speech model, supports audio and text input, and outputs text and speech replies in realtime. **This API is designed for server-side integration; browsers cannot connect directly.** For security reasons, the gateway validates and rejects connections that carry an `Origin` header, rejects the `openai-insecure-api-key` subprotocol, and accepts the key only through the standard `Authorization` header. A browser-initiated WebSocket automatically attaches an `Origin` header and is therefore rejected. To do realtime conversation in a frontend, establish the connection to the gateway from your own server and forward audio and results between your frontend and server. ## Quick Start ### Connection endpoint ```text theme={null} wss://aihubmix.com/v1/realtime?model=gpt-realtime-2.1 ``` * `model=gpt-realtime-2.1`: **required**. The model is fixed by the URL parameter at connection time and cannot be changed during the session (see the constraints below). * **Note the difference from transcription**: the conversation endpoint does **not** take `intent=transcription`. ### Authentication Pass the key in a standard HTTP header during the handshake: ```text theme={null} Authorization: Bearer $AIHUBMIX_API_KEY ``` ### Audio format requirements Audio input and output currently support a single format. Convert your audio before sending: * **Encoding**: PCM16 (16-bit signed integer, little-endian) * **Sample rate**: 24000 Hz * **Channels**: mono That is `audio/pcm@24000`. Declaring any other format (such as G.711/ยต-law) for input or output is rejected and the session is closed. Unlike transcription, conversation sessions **do** support voice activity detection (turn\_detection / VAD). When enabled, the model decides when an utterance ends and triggers a reply; when disabled (set to `null`), you control when to commit audio and when to request a reply. Choose either as needed. ## Session configuration (session.update) After the connection is established, the client can send one `session.update` frame to configure conversation parameters such as the voice, system instructions, and whether VAD is enabled. The conversation model is already anchored by the connection URL, so **you can start talking without sending `session.update`**. Send it when you need a custom voice or instructions. ```json theme={null} { "type": "session.update", "session": { "type": "realtime", "instructions": "You are a helpful voice assistant. Keep answers concise.", "audio": { "input": { "format": { "type": "audio/pcm", "rate": 24000 }, "turn_detection": { "type": "server_vad" } }, "output": { "format": { "type": "audio/pcm", "rate": 24000 }, "voice": "alloy" } } } } ``` ### Configuration parameters Session type. For conversation it is `realtime`. System instructions that set the model's role, tone, and answer constraints. Output modalities, either `["audio"]` or `["text"]`: with `["audio"]` (the default) the model outputs speech and the reply text arrives through `response.output_audio_transcript.delta`; with `["text"]` it outputs text only, delivered through `response.output_text.delta`. Any other combination (for example `["audio", "text"]`) is rejected with an `error` event. Input audio format, fixed at `{ "type": "audio/pcm", "rate": 24000 }`. Voice activity detection. Pass `{ "type": "server_vad" }` to enable automatic turn detection; pass `null` to disable it and commit audio and request replies manually from the client. Output audio format, fixed at `{ "type": "audio/pcm", "rate": 24000 }`. Voice for the reply audio. **It cannot be changed once the first reply starts**: after the session enters a generating state, a `voice` sent again is ignored (other settings still apply). Set the voice before requesting the first reply if you need a specific one. The following capabilities are **not supported in this release** and close the session when configured (close code `1008`): enabling inline transcription inside a conversation session (`audio.input.transcription`, reason `input_transcription_not_supported`), injecting audio (reason `item_audio_not_supported`) or images (reason `image_input_not_supported`) through conversation items, and any content item type other than text (reason `unsupported_content_part`). Send all audio through the `input_audio_buffer.append` channel. ## Sending input ### Sending audio Cut PCM16 audio into small chunks (for example one chunk per 100ms), base64-encode them, and send them continuously with the `input_audio_buffer.append` event: ```json theme={null} { "type": "input_audio_buffer.append", "audio": "" } ``` With VAD enabled, the model detects the end of an utterance and triggers a reply automatically. With VAD disabled, commit and request a reply manually after sending a segment of audio: ```json theme={null} { "type": "input_audio_buffer.commit" } { "type": "response.create" } ``` ### Sending text You can also inject a text message directly and request a reply: ```json theme={null} { "type": "conversation.item.create", "item": { "type": "message", "role": "user", "content": [{ "type": "input_text", "text": "Introduce yourself in one sentence." }] } } { "type": "response.create" } ``` ## Receiving replies The server keeps pushing events. Key event types: Confirmation that the session was created or its configuration updated. You can start sending audio and text once `session.created` arrives. A conversation item has been written: each user input and each model reply adds one item. With VAD enabled, the server has detected that the user started or stopped speaking. A `speech_started` event usually means the user is interrupting the model; see [Interruption and truncation](#interruption). A reply has started generating. The reply's output item has started and finished. The `item.id` field in the `added` event is the conversation item ID you reference when truncating audio later. An **incremental** chunk of the reply audio (base64-encoded PCM16) and its completion marker, which you can play as it arrives. The **incremental** transcript matching the reply audio sentence by sentence, plus its completion marker; the `delta` field holds the newly added text. **When audio output is enabled, take the reply text from this event**, for example to show captions while the audio plays. An **incremental** chunk of a text-only reply and its completion marker, emitted only when you set the output modality to text only (`output_modalities: ["text"]`). A reply has finished. This event carries the token usage for the turn (`usage`), which is what billing is based on. Confirmation that the truncation request took effect; see [Interruption and truncation](#interruption). An error event, with an error code and description. A problem with the request itself (for example an invalid `output_modalities` value) returns a single `error` event and the session stays usable; policy issues (such as changing the model or running out of balance) close the session. **Pick the right event for the reply text.** By default (output includes audio) the model pushes only `response.output_audio_transcript.delta` and **does not** push `response.output_text.delta`; set the output modality to text only and the text moves to `response.output_text.delta`. Listen for both in either mode so you never miss text (see [run output](#run-output) below).

Interruption and truncation

When the user starts speaking while the model is talking, content that has been generated but not yet played back conflicts with the user's next sentence. On a WebSocket connection the client handles playback, so the client also completes the cleanup after an interruption. With VAD enabled, the server pushes `input_audio_buffer.speech_started` once it detects that the user has started speaking. On receiving that event, the client should: 1. **Stop local playback immediately** and record how far into the reply it had played (in milliseconds). 2. Send `conversation.item.truncate` to remove the unplayed audio from the conversation, so the model does not treat it as spoken in the next turn. ```json theme={null} { "type": "conversation.item.truncate", "item_id": "item_ABC123", "content_index": 0, "audio_end_ms": 1500 } ``` * `item_id`: the conversation item ID of this reply, taken from `item.id` in the `response.output_item.added` event. * `content_index`: the index of the audio content part, always `0`. * `audio_end_ms`: how much audio to keep, in milliseconds, based on where playback actually reached. The server replies with `conversation.item.truncated` once the request is processed. Truncation only affects this reply's audio and its transcript; the session itself is unaffected and you can continue with the next turn. With the OpenAI SDK, call `conn.conversation.item.truncate(item_id=..., content_index=0, audio_end_ms=...)`. With VAD disabled (for example push-to-talk), pressing the button is the interruption: send `response.cancel` to cancel the in-progress reply and truncate as described above, then on release send `input_audio_buffer.append`, `input_audio_buffer.commit`, and `response.create` in that order. ## Complete examples Three approaches are shown below; pick one: * **Official OpenAI SDK (recommended)**: no need to hand-write WebSocket frames; point `websocket_base_url` (the SDK's WebSocket base URL parameter) at the gateway and reuse the official library. * **Official OpenAI Agents SDK**: the official agent framework's realtime voice form; swap the `url` in `model_config` for the gateway address. * **Raw websockets**: no SDK, exchange frames directly over the protocol. Fewest dependencies and easiest to debug. **Why pass the model name at connection time?** The AiHubMix gateway needs the `model` at the moment of the WebSocket handshake to select the model provider, authenticate, and reserve quota, while `session.update` only arrives after the handshake completes. So with an SDK you must pass `model` explicitly to `connect()` (the SDK puts it into the URL query); without it the gateway **rejects the connection during the handshake** and no connection is established. Unlike transcription, the conversation endpoint does **not** need `intent=transcription`. ```python Python (OpenAI SDK) theme={null} # Dependency: pip install "openai[realtime]" import asyncio import base64 from openai import AsyncOpenAI # Reusing the official SDK requires only two adaptations: # 1) point websocket_base_url at the AiHubMix gateway (instead of OpenAI's default address) # 2) pass model explicitly to connect(), required for the gateway handshake (the conversation endpoint needs no intent) client = AsyncOpenAI( api_key="sk-***", # Replace with your AiHubMix API key websocket_base_url="wss://aihubmix.com/v1", ) async def main(): async with client.realtime.connect(model="gpt-realtime-2.1") as conn: # 1) Configure the session: system instructions + voice + VAD (automatic turn detection) await conn.session.update(session={ "type": "realtime", "instructions": "You are a helpful voice assistant. Keep answers concise.", "audio": { "input": { "format": {"type": "audio/pcm", "rate": 24000}, "turn_detection": {"type": "server_vad"}, }, "output": { "format": {"type": "audio/pcm", "rate": 24000}, "voice": "alloy", }, }, }) # 2) Send a text message and request a reply (for audio input see append/commit in the raw example) await conn.conversation.item.create(item={ "type": "message", "role": "user", "content": [{"type": "input_text", "text": "What is the capital of France?"}], }) await conn.response.create() # 3) Receive this turn's reply: text arrives as transcript deltas, speech as audio deltas async for event in conn: if event.type == "response.output_audio_transcript.delta": print(event.delta, end="", flush=True) # reply text (when output includes audio) elif event.type == "response.output_audio_transcript.done": print() # this turn's transcript finished elif event.type == "response.output_text.delta": print(event.delta, end="", flush=True) # text-only output elif event.type == "response.output_text.done": print() elif event.type == "response.output_audio.delta": pass # base64 PCM16 audio chunk, decode to play elif event.type == "input_audio_buffer.speech_started": pass # user interruption: stop playback, then call conn.conversation.item.truncate(...) elif event.type == "response.done": print("\n[turn complete]", event.response.usage) break elif event.type == "error": print("\n[error]", event.to_dict()) break asyncio.run(main()) ``` ```python Python (websockets) theme={null} import asyncio import base64 import json import websockets API_KEY = "sk-***" # Replace with your AiHubMix API key URL = "wss://aihubmix.com/v1/realtime?model=gpt-realtime-2.1" async def main(): # websockets >= 13 uses additional_headers; older versions use extra_headers async with websockets.connect( URL, additional_headers={"Authorization": f"Bearer {API_KEY}"} ) as ws: # 1) Configure the session (VAD off, control the reply timing manually) await ws.send(json.dumps({ "type": "session.update", "session": { "type": "realtime", "instructions": "You are a helpful voice assistant.", "audio": { "input": { "format": {"type": "audio/pcm", "rate": 24000}, "turn_detection": None, }, "output": { "format": {"type": "audio/pcm", "rate": 24000}, "voice": "alloy", }, }, }, })) # 2) Read a local raw PCM16 / 24kHz / mono file, send it in chunks, then commit + request a reply async def send_audio(): with open("audio_pcm16_24k.raw", "rb") as f: pcm = f.read() chunk = 24000 * 2 // 10 # 100ms = sample rate x 2 bytes / 10 for i in range(0, len(pcm), chunk): await ws.send(json.dumps({ "type": "input_audio_buffer.append", "audio": base64.b64encode(pcm[i:i + chunk]).decode(), })) await asyncio.sleep(0.1) # simulate realtime pacing # With VAD off, commit and request a reply manually await ws.send(json.dumps({"type": "input_audio_buffer.commit"})) await ws.send(json.dumps({"type": "response.create"})) asyncio.create_task(send_audio()) # 3) Receive the reply async for msg in ws: evt = json.loads(msg) etype = evt.get("type", "") if etype == "response.output_audio_transcript.delta": print(evt.get("delta", ""), end="", flush=True) # reply text (when output includes audio) elif etype == "response.output_audio_transcript.done": print() # this turn's transcript finished elif etype == "response.output_text.delta": print(evt.get("delta", ""), end="", flush=True) # text-only output elif etype == "input_audio_buffer.speech_started": pass # user interruption: stop playback, then send conversation.item.truncate elif etype == "response.output_audio.delta": pass # base64 PCM16 audio chunk, decode to play elif etype == "response.done": print("\n[turn complete]", evt.get("response", {}).get("usage")) break elif etype == "error": print("\n[error]", evt.get("error")) break asyncio.run(main()) ``` ```python Python (Agents SDK) theme={null} # Dependency: pip install openai-agents import asyncio from agents.realtime import ( OpenAIRealtimeWebSocketModel, RealtimeAgent, RealtimeSession, ) agent = RealtimeAgent( name="Assistant", instructions="You are a helpful voice assistant. Keep answers concise.", ) async def main(): session = RealtimeSession( OpenAIRealtimeWebSocketModel(), agent, None, model_config={ "api_key": "sk-***", # Replace with your AiHubMix API key "url": "wss://aihubmix.com/v1/realtime?model=gpt-realtime-2.1", }, run_config={ "model_settings": { "modalities": ["audio"], "output_audio_format": "pcm16", # Must disable: the input audio transcription this API does not support, # and leaving it on gets the session closed with 1008 "input_audio_transcription": None, }, }, ) async with session: await session.send_message("What is the capital of France?") async for event in session: if getattr(event, "type", "") == "raw_model_event": data = getattr(event, "data", None) if getattr(data, "type", "") == "transcript_delta": print(getattr(data, "delta", ""), end="", flush=True) elif getattr(data, "type", "") == "turn_ended": print("\n[turn complete]") return asyncio.run(main()) ``` ```bash Connection test (wscat) theme={null} # Use wscat to quickly verify connectivity and authentication (requires npm i -g wscat) wscat -c "wss://aihubmix.com/v1/realtime?model=gpt-realtime-2.1" \ -H "Authorization: Bearer $AIHUBMIX_API_KEY" # Once connected, paste a session.update frame, then send a text message + response.create to start ``` To convert any audio into the raw PCM format this API requires, use ffmpeg: ```bash theme={null} ffmpeg -i input.mp3 -f s16le -acodec pcm_s16le -ac 1 -ar 24000 audio_pcm16_24k.raw ``` ### Reusing official examples Most realtime conversation examples OpenAI publishes rely only on the SDK's base URL parameter, so you can reuse them by pointing the address at the AiHubMix endpoint: | Official example | Reusable with address change only | What to change | | ------------------------------------------------------------------------------------ | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Official Python SDK realtime conversation example | Yes | Set `websocket_base_url` to `wss://aihubmix.com/v1` and pass `model` to `connect()` | | Official Node / JS SDK realtime conversation example | Yes | Set `baseURL` to `https://aihubmix.com/v1` (the SDK switches it to `wss` and builds `/realtime?model=...`) | | Official Agents SDK realtime voice example | Yes | Set `model_config.url` to `wss://aihubmix.com/v1/realtime?model=gpt-realtime-2.1` and disable the input audio transcription it enables by default | | Official browser demos (such as the realtime console and the multi-agent voice demo) | No | These demos connect directly from the browser and are rejected by the gateway's `Origin` check; they also depend on server-issued ephemeral credentials, while this API exposes server-side WebSocket only |

Run output (measured in production)

The following is the real output of the OpenAI SDK example against the `aihubmix.com` production environment with model `gpt-realtime-2.1`. The session enables `server_vad`, and both prompts and audio are English. **Text input** ```text theme={null} # Send the text "What is the capital of France?" [turn 1, text] completed in 2.33s | 2.70s of PCM16 audio output reply: The capital of France is Paris. usage: input_tokens 29 (text 29) / output_tokens 81 (audio 54 + text 27, including reasoning 8) ``` **Audio input** ```text theme={null} # Send 6.3s of English speech in chunks; the model detects the turn end and replies [turn 2, audio] completed in 9.59s | 4.35s of PCM16 audio output reply: Everything sounds good on my side, and I'm ready to keep this conversation going. usage: input_tokens 117 (audio 67 + text 50) / output_tokens 129 (audio 87 + text 42, including reasoning 11) VAD events: input_audio_buffer.speech_started / input_audio_buffer.speech_stopped ``` **Measured difference between two configurations** | Configuration | Measured result | | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `turn_detection: {"type": "server_vad"}` | After the audio is sent you receive `input_audio_buffer.speech_started` and `speech_stopped`, and the model detects the turn end and starts replying without a manual `commit` and `response.create` | | `output_modalities: ["text"]` | The reply text moves to `response.output_text.delta`, no audio output events occur for the turn, and `audio_tokens` is 0 in usage | Measured confirmation: by default (output includes audio) the reply text arrives character by character only from `response.output_audio_transcript.delta`, and `response.output_text.delta` never appears; `response.done` carries the turn's `usage`; the handshake takes about 2 to 4 seconds, the quota reservation cost of establishing the session. ## Billing * **Billed by token**: a conversation session returns the token usage for each turn (`usage`) with the `response.done` event, and billing is based on it. Usage is metered separately by component, including **audio input, audio output, text input, and text output**. The unit price of each component follows the live listed price on the [model detail page](https://aihubmix.com/model/gpt-realtime-2.1). * **Settled as you go**: this is a long-lived connection, and charges are deducted in realtime with each turn as the session runs, with no single settlement at the end. Establishing a session first makes a **quota reservation** of about one minute of usage (an admission check only, not an actual charge), and the remaining reservation is released when the session ends. **Your account's available balance must therefore cover at least about one minute of usage for a session to be established.** * Each realtime conversation billing record can be reviewed entry by entry in [Usage and billing](https://aihubmix.com). ## Limits and constraints 1. **Session duration**: a WebSocket connection lasts at most **62 minutes**, after which the server closes it (close code `1000`, reason `session_duration_limit`); reconnect in segments if you need longer. 2. **Idle disconnect**: when neither the client nor the model has any activity for **5 minutes**, the server closes the session (close code `1008`, reason `idle_timeout`). Activity from either side resets the timer, so a long reply that keeps streaming is not interrupted. 3. **Insufficient balance**: **when establishing a session**, if the available balance does not cover the roughly one-minute reservation, the handshake is rejected outright (HTTP 403) and no session is established; **during a session**, if the balance runs out, the established connection is closed immediately. 4. **Server-side only**: direct browser connections are not supported (the `Origin` header is validated); integrate from your server. 5. **Model locked**: the model is fixed in the connection URL, and changing it with `session.update` during the session is rejected and closes the session. 6. **Format locked**: audio input and output support only `audio/pcm@24000` mono; other formats are rejected. 7. **Voice locked**: `voice` cannot be changed after the first reply starts; set it before requesting the first reply. 8. **Not yet supported**: inline transcription inside a conversation session, injecting audio or image content through conversation items, and any content item type other than text. 9. **One reply at a time**: a session allows only one in-progress reply; sending another `response.create` before the current turn finishes is rejected and closes the session (close code `1008`, reason `response_already_active`). ## Common errors | Scenario | Close code / status | Description | | --------------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------- | | Service not enabled | HTTP 403 `realtime_disabled` | Realtime conversation is not open for this environment | | `Origin` header present / direct browser connection | Handshake rejected | Use a server-side connection | | Insufficient balance at connect time | HTTP 403 `insufficient_user_quota` | The balance cannot cover the roughly one-minute reservation; top up and retry | | Changing the model mid-session | `1008` `model_override_forbidden` | The model can only be specified in the connection URL | | Audio format not PCM | `1008` `audio_format_unsupported` | Convert to `audio/pcm@24000` | | Changing the voice after the first reply | `voice` field ignored | Set `voice` before requesting the first reply | | Enabling inline transcription | `1008` `input_transcription_not_supported` | Not supported in conversation sessions in this release | | Injecting audio through conversation items | `1008` `item_audio_not_supported` | Send audio through `input_audio_buffer.append` | | Injecting images through conversation items | `1008` `image_input_not_supported` | Not supported in this release | | Conversation item with a non-text content type | `1008` `unsupported_content_part` | Conversation items support text content only | | Invalid `output_modalities` value | `error` event `invalid_value` | Supported values are `["audio"]` and `["text"]`; the session stays open | | Session reaches its maximum duration | `1000` `session_duration_limit` | The 62-minute limit was reached; reconnect to continue | | Both sides idle for 5 minutes | `1008` `idle_timeout` | Activity from either side resets the timer | | Another request while a reply is in progress | `1008` `response_already_active` | Wait for `response.done` before sending `response.create` | | Duplicate `event_id` | `1008` `duplicate_event_id` | Each client event's `event_id` must be unique within the session | | `response.conversation` set to a non-default value | `1008` `conversation_mode_not_supported` | Only the default conversation mode is supported | | Client sends a server-only event | `1008` `client_forged_lifecycle_event` | In the `response.*` namespace clients may only send `response.create` and `response.cancel` | | Balance exhausted mid-session | Connection closed | Top up and reconnect | *** Last updated: 2026-09-21