Transcribe a conversation as it happens, for a voice agent, over one WebSocket connection. Open the connection and send StartRecognition, then stream audio as binary messages. Every other message is a JSON object that names itself in its message field.

Three groups of server messages carry the conversation itself, and are the ones your agent logic acts on:

Group	Messages	What they mean
Speech signals	`SpeechStarted`, `SpeechEnded`	Voice activity. Sent in `vad` mode only, and not turn boundaries — a mid-sentence pause ends speech without ending the turn.
Segments	`AddPartialSegment`, `AddSegment`	The transcript. Each partial is a snapshot that replaces the last; `AddSegment` is final and will not change.
Turn signals	`StartOfTurn`, `EndOfTurn`	Where a speaker's turn starts and ends. `EndOfTurn` is the turn's last message, and the cue for your agent to respond.

Pass AddSegment downstream to your LLM, and use AddPartialSegment only for display. The other server messages manage the session rather than the conversation.

Handshake

GET wss://global.rt.speechmatics.com/v2/agent/

Routes each connection to the nearest region, for the lowest latency. To keep data in one region instead, connect to that region's endpoint, such as wss://eu.rt.speechmatics.com/v2/agent. See Supported endpoints.

Authentication

Any one of:

apiKey Authorization: Bearer <token>
Your API key, for server-side connections.
temporaryKey ?jwt=<key>
A Realtime (type=rt) temporary key, for connections from a browser, which cannot set the Authorization header — and should not hold your long-lived API key. For example, wss://global.rt.speechmatics.com/v2/agent?jwt=<temporary-key>.

Responses

Switching Protocols. Connected: send StartRecognition to begin.

A successful handshake request:

GET /v2/agent HTTP/1.1
Host: eu.rt.speechmatics.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: ujRTbIaQsXO/0uCbjjkSZQ==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Authorization: Bearer wmz9fkLJM6U5NdyaG3HLHybGZj65PXp
User-Agent: Python/3.12 websockets/15.0

And its response:

HTTP/1.1 101 Switching Protocols
Connection: upgrade
Upgrade: WebSocket
Sec-WebSocket-Accept: 87kiC/LI5WgXG52nSylnfXdz260=

Method Not Allowed. The request method is not GET.

Send client → server

StartRecognition

Initiates a new recognition session.

audio_format AudioFormat
encoding "pcm_f32le" | "pcm_s16le"
sample_rate 8000 | 16000
type "raw"
message "StartRecognition"
transcription_config TranscriptionConfigno additional propertiesextends SpeakerDiarizationConfigRule
additional_vocab WordListone of optional: List of custom words or phrases that should be recognized. Alternative pronunciations can be specified to aid recognition.
Word Word≥ 1 charactersmatches /\S/ optional
Option 2 objectno additional properties optional
content Word≥ 1 charactersmatches /\S/
sounds_like PronunciationList≥ 1 items optional
diarization "none" | "speaker" optional: Specify whether speaker labels are added to the transcript. The default is none.

none: no diarization labels are added.

speaker: speaker attribution is performed based on acoustic matching.
domain string≥ 1 characters optional: Request a specialized model based on 'language' but optimized for a particular field, e.g. 'finance' or 'medical'.
emit_sentences booleandefault: false optional: Whether or not to split multi-sentence segments on sentence boundaries, so that each segment contains a single sentence. The default is false.
enable_partials booleandefault: false optional: Whether or not to send partials as well as finals.
language string≥ 1 characters: Language model to process the audio input, normally specified as an ISO language code
model "linden-1": The model to use for transcription. Agent STT is powered by linden-1.
output_locale string optional: Language locale to be used when generating the transcription output, normally specified as an ISO language code
speaker_diarization_config object | nullno additional properties optional
get_speakers boolean optional: If true, speaker identifiers will be returned at the end of transcript.
max_speakers integer≥ 2 optional: Specifies the maximum number of speakers allowed in a single audio stream. If not provided, the number of speakers is unrestricted.
prefer_current_speaker boolean optional: If true, the diarization engine will prefer to stick with the currently active speaker if good enough, even if not the best. This is useful for cases where we can flip incorrectly between similar speakers during a single speaker section.
speaker_sensitivity number≥ 0, ≤ 1 optional: Controls the number of unique speakers detected; a higher value is likely to result in more unique speakers being identified in the output. You may want to increase this value if you are seeing fewer speaker labels than expected. The default is 0.5.
speakers SpeakersInputItem[]no additional properties optional
label unknown`not` has no display form: Speaker label, which must not match the format used internally (e.g. S1, S2, etc).
speaker_identifiers Array<string<bytes>>≥ 1 itemsunique items
transcript_filtering_config object | nullno additional properties optional: Configuration for applying filtering to the transcription
remove_disfluencies boolean optional: If true, words that are identified as disfluencies will be removed from the transcript. If false, they are tagged in the transcript as 'disfluency'.
replacements object[] optional
from string: The pattern to be replaced.
to string: The word or phrase to replace the 'from' value.
turn_config TurnConfigno additional properties optional
turn_detection_mode "vad" | "external"default: "vad" optional: How the end of a speaker turn is detected. The default is vad.

vad: turn ends are detected automatically from voice activity.

external: turn ends are signalled by the client with the ForceEndOfUtterance message.

AddAudio

A binary chunk of audio. The server confirms receipt by sending an AudioAdded message.

application/octet-stream

AddAudio

EndOfStream

Declares that the client has no more audio to send.

last_seq_no integer
message "EndOfStream"

SetRecognitionConfig

Allows the client to re-configure the recognition session.

The language field can be included in SetRecognitionConfig, but its value must match the language specified in the initial StartRecognition message. Attempting to change the language or any other transcription configuration parameter not listed below will result in an error.

message "SetRecognitionConfig"
transcription_config MidSessionTranscriptionConfigno additional properties
enable_partials booleandefault: false optional: Whether or not to send partials as well as finals.
language string≥ 1 characters optional: Language model to process the audio input, normally specified as an ISO language code

GetSpeakers

Requests any detected speaker identifiers to be returned.

final boolean optional
message "GetSpeakers"

ForceEndOfUtterance

Closes the current speaker turn. The transcript up to timestamp is flushed as an AddSegment, followed by EndOfTurn.

Supported only with turn_config.turn_detection_mode: external, where the client detects turn ends itself. In the default vad mode the service closes turns from voice activity.

message "ForceEndOfUtterance"
timestamp number<float>≥ 0: Timestamp of the audio data that corresponds to the force end of utterance request. It's the number of seconds since the beginning of the audio.

Receive server → client

RecognitionStarted

Server response to StartRecognition, acknowledging that a recognition session has started.

id string
language_pack_info LanguagePackInfono additional properties optional: Properties of the language pack.
adapted boolean optional: Whether or not language model adaptation has been applied to the language pack.
itn boolean optional: Whether or not ITN (inverse text normalization) is available for the language pack.
language_description string optional: Full descriptive name of the language, e.g. 'Japanese'.
word_delimiter string: The character to use to separate words.
writing_direction "left-to-right" | "right-to-left" optional: The direction that words in the language should be written and read in.
message "RecognitionStarted"
orchestrator_version string optional

AudioAdded

Server response to AddAudio, indicating that audio has been added successfully.

When clients send audio faster than real-time, the server may read data slower than it's sent. If binary AddAudio messages exceed the server's internal buffer, the server will process other WebSocket messages until buffer space is available. Clients receive AudioAdded responses only after binary data is read. This can fill TCP buffers, potentially causing WebSocket write failures and connection closure with prejudice. Clients can monitor the WebSocket's bufferedAmount attribute to prevent this.

message "AudioAdded"
seq_no integer≥ 0

AddPartialSegment

An interim, best-guess preview of the segment currently being built. Subject to change.

message "AddPartialSegment"
metadata SegmentMetadatano additional properties
end_time number<float>: Segment end, in seconds from session start.
start_time number<float>: Segment start, in seconds from session start.
segment Segmentno additional properties
speaker string optional: Speaker label (e.g. S1), present only when diarization attributed one.
transcript string: The rendered segment text.

AddSegment

A finalized, stable segment, sent when a segment boundary is reached. It will not change.

message "AddSegment"
metadata SegmentMetadatano additional properties
end_time number<float>: Segment end, in seconds from session start.
start_time number<float>: Segment start, in seconds from session start.
segment Segmentno additional properties
speaker string optional: Speaker label (e.g. S1), present only when diarization attributed one.
transcript string: The rendered segment text.

SpeechStarted

Sent when voice activity detection detects the onset of speech.

message "SpeechStarted"
metadata SpeechStartedMetadatano additional properties
start_time number<float>: Time speech began, in seconds from session start.

SpeechEnded

Sent when voice activity detection detects the end of speech.

message "SpeechEnded"
metadata SpeechEndedMetadatano additional properties
end_time number<float>: Time speech ended, in seconds from session start.

StartOfTurn

Sent when turn detection marks the start of a speaker turn.

message "StartOfTurn"
metadata StartOfTurnMetadatano additional properties
start_time number<float>: Time the turn began, in seconds from session start.

EndOfTurn

Sent when turn detection marks the end of a speaker turn. It is the last message of a turn.

message "EndOfTurn"
metadata EndOfTurnMetadatano additional properties
end_time number<float>: Time the turn ended, in seconds from session start.

EndOfTranscript

Server response to EndOfStream, sent after the server has finished sending all segment messages.

message "EndOfTranscript"

Error

Error messages sent from the server to the client. After any error, transcription is terminated and the connection is closed.

message "Error"
reason string
type enum10 values

Warning

Warning messages sent from the server to the client.

duration_limit number≥ 0 optional
message "Warning"
reason string
type enum"duration_limit_exceeded""idle_timeout""session_timeout""add_audio_after_eos""speaker_id""unsupported_language" optional

Info

Additional information sent from the server to the client.

message "Info" optional
quality "broadcast" optional
reason string optional
type "recognition_quality" | "concurrent_session_usage" optional

SpeakersResult

Server response to GetSpeakers, returning any detected speaker identifiers.

message "SpeakersResult"
speakers object[]
label string≥ 1 characters: Speaker label.
speaker_identifiers Array<string<bytes>>≥ 1 itemsunique items

Connection close

An Error message can be followed by a WebSocket close frame. Its code, and the reason it carries:

Close code	Reason
1003	`protocol_error`
1008	`policy_violation`
1011	`internal_error`
4001	`not_authorised`
4003	`not_allowed`
4004	`invalid_model`
4005	`quota_exceeded`
4006	`timelimit_exceeded`
4013	`job_error`