How to Use OpenAI GPT-Realtime-2 API: Step by Step Guide

OpenAI recently launched GPT-Realtime-2, a speech-to-speech reasoning model that simplifies building real-time voice applications. Alongside it, OpenAI released two specialized models: GPT-Realtime-Whisper for transcription and GPT-Realtime-Translate for live translation. This complete beginner guide and step by step tutorial walks you through exactly how to connect to each API using Python WebSockets, send audio, and handle responses. By the end of this tutorial, you will have tested all three models with working code and understand when to use each one.

What is the GPT-Realtime-2 API?

GPT-Realtime-2 is OpenAI's speech-to-speech reasoning model. It replaces the separate STT-to-LLM-to-TTS pipeline with a single model that understands audio directly, reasons about it, and responds with natural speech. It supports a 128K context window and controllable reasoning effort, making it suitable for long voice sessions.

The Realtime API actually covers three distinct models, each designed for a specific job:

This beginner guide covers the complete setup process. Before you begin, make sure you have:

Step 1: Choose Between WebRTC and WebSocket

OpenAI provides two transport paths for the Realtime API. The right choice depends entirely on your client environment:

• WebRTC — Use for browser and mobile clients. WebRTC handles jitter buffering, audio transport, and media stream management natively. For browser builds, use ephemeral client secrets so your API key never reaches the frontend.
• WebSocket — Use for server-side applications. WebSockets make sense when your backend already receives raw audio from a telephony provider or media pipeline. This path shows event names directly, making model differences visible in the code.

For this complete tutorial, all three tests use WebSockets since we are connecting from a Python server environment.

Step 2: Set Up Python Environment and Dependencies

The full code for all scripts is available on GitHub. Clone the repository first, then install the required dependencies:

The websocket-client library handles the WebSocket connection. sounddevice captures microphone audio, numpy converts the audio buffer, and python-dotenv loads the API key from a .env file.

Step 3: Configure Authentication and WebSocket URL

Create a .env file at the root of your project and add your OpenAI API key:

Server-side connections use an Authorization: Bearer header on the WebSocket handshake. Each model uses a different WebSocket URL:

Load the API key in each script using python-dotenv:

Step 4: Test Real-Time Audio Transcription with GPT-Realtime-Whisper

Transcription is the simplest use case. If your output is only text, the transcription model is all you need. GPT-Realtime-Whisper takes audio in and emits transcript deltas. It does not reason, call tools, or speak back, which makes it more cost-effective than using the full GPT-Realtime-2 model.

The key configuration field is session.type: "transcription". This tells the API to skip assistant responses and emit transcript events only. Use 24 kHz PCM16 mono audio, encoded as base64:

Unlike the voice agent session, the transcription script commits the input buffer manually on a timer instead of using server_vad turn detection. Empty commits raise input_audio_buffer_commit_empty errors, so the script only commits after real audio has been sent. Transcript deltas arrive word by word in real time, typically within 3 to 4 seconds after speech starts.

Step 5: Test Real-Time Translation with GPT-Realtime-Translate

Live speech translation looks similar to transcription but uses a separate endpoint. The translation endpoint removes the assistant response loop entirely. Translation sessions have no assistant turn loop and no response.create. The model works as a live interpreter, not a conversational agent.

The model supports more than 70 input languages and 13 output languages. You set the target language with session.audio.output.language. Source language detection is automatic. Key limitations: no custom prompting, no voice selection, and no domain glossaries.

Translated audio arrives on session.output_audio.delta events, and the audio bytes are in event["delta"], not event["audio"]. The source and translated transcripts arrive separately:

For short English-to-Spanish phrases, translated audio begins before the source utterance finishes. More distant language pairs can wait longer for context. One edge case: if the source audio is already in the target language, the model may produce silence rather than pass it through.

Step 6: Build a Voice Assistant with GPT-Realtime-2

This is the main GPT-Realtime-2 test: a voice agent that listens, speaks, keeps context, and can call tools. It is a speech-to-speech reasoning model with a 128K context window. The session configuration uses semantic_vad turn detection, which looks at speech cues rather than silence alone:

When the user's transcript completes, the client creates the assistant response. Assistant audio arrives on response.output_audio.delta events:

Handling Barge-In and Interruptions

The interruption sequence is critical for a natural voice experience. When the user speaks over the assistant, the server sends input_audio_buffer.speech_started. The client must stop playback, record how much audio it played, and send conversation.item.truncate with audio_end_ms to tell the server where it was cut off:

Cost and Latency Overview

Pricing splits into two groups: GPT-Realtime-2 is billed by token, while translation and transcription are billed by audio minute:

Voice agents are harder to estimate because audio tokens accumulate on both sides of the conversation. Session length, speaking ratio, reasoning effort, and context size all matter. Prompt caching can reduce cost when earlier conversation turns stay stable.

API Limits, Audio Requirements, and Troubleshooting

Final Thoughts

The same pattern shows up across all three tests: each job has its own model and endpoint. That split affects what the model can do, how it bills, and how much client code you need to own. GPT-Realtime-Whisper covers live text, GPT-Realtime-Translate covers direct speech translation, and GPT-Realtime-2 covers full assistant behavior with speech, reasoning, and context.

The code does not show one model replacing the others. It shows that realtime voice apps depend on session design. Your starting point should be the smallest model that matches the job, with the remaining engineering time spent on audio quality, turn-taking, reconnects, and client state management.