Skip to main content
Media Streaming provides instant access to your raw call media. With it, you can deliver exceptional customer experiences—from unlocking new insights through sentiment analysis to providing fast resolutions using AI solutions. You can also bring your own AI engine and use real-time media streaming to connect it directly to call control, enabling custom AI-powered voice applications. When your call is established, Telnyx takes the call media and forks it, so recipients receive the call simultaneously. The Telnyx network ensures that call media can be duplicated, delivered, analyzed, and returned in real-time. The secondary delivery recipient never occupies the call stream, so you never have to worry about degraded quality or dropped connections. This guide covers how to set up media streaming over Websockets.
streaming_start is synchronous and might take longer to respond. answer and dial requests with stream_url can also be synchronous. We advise not using the 500 ms retry rule for these requests, and waiting for the response before retrying. See Command Retries.

Requesting streaming using Dial Command

The requesting dial command can be extended in the following way to request streaming using WebSockets:
The following additional attributes need to be added to the request:
  • stream_url - the destination address when the stream is going to be delivered.
  • stream_track - specifies which track should be streamed, with possible options:
    • inbound_track (default)
    • outbound_track
    • both_tracks
As a response, a regular confirmation is sent:

Requesting streaming using Answer Command

Using the same attributes as above, streaming can be requested while answering the call:
In this case, confirmation that the call has been answered includes streaming details:

Verify the WebSocket connection

Telnyx signs every media stream WebSocket connection, so you can confirm that a connection to your stream_url comes from Telnyx. The signature uses your account’s Ed25519 key pair, the same one Telnyx uses to sign webhooks. Each connection request includes two headers:
  • telnyx-signature-ed25519: the Base64-encoded Ed25519 signature.
  • telnyx-timestamp: the Unix time, in seconds, when the request was signed.
The signature covers the string {telnyx-timestamp}|{stream_url}, where stream_url is the exact URL you set when starting the stream, including any query string. To check it, verify the Base64-decoded telnyx-signature-ed25519 header over that exact string with your account’s Ed25519 public key, available in the Mission Control Portal.

Streaming process flow

When the WebSocket connection is established, the following event is being sent:
Before the stream begins, the streaming.started webhook is sent:
An event over WebSockets which contains information in the mediaFormat section about the encoding and stream_id that identifies a particular stream:
The following media events follow the start event:
The payload contains a base64-encoded RTP payload (no headers).
The order of events is not guaranteed. Use sequence_number or chunk to reorder them; see Frame numbering.
When the call ends, the streaming.stopped webhook is sent:
And the stop event over WebSockets connection is sent:
Currently only one streaming/fork operation is supported per call. In case of requesting media forking the WebSocket stream will be stopped and replaced by a RTP connection.

Stopping a stream

After streaming_stop or hangup, Telnyx normally delivers the media already queued for your WebSocket, then sends the stop frame and closes the connection with code 1000. If the connection is already failing, the stop frame may not arrive. The streaming.stopped webhook can arrive before the WebSocket closes. Treat the stop frame or the WebSocket closing, whichever comes first, as the end of the stream. Read from the WebSocket at least as fast as real time. If your application reads more slowly, queued media builds up and latency grows. A WebSocket that accepts no data for about 5 seconds is treated as disconnected; see Reconnects.

Bidirectional media streaming

Sending RTP stream

The RTP stream can be sent to the call using websocket. The functionality can be enabled by setting stream_bidirectional_mode to rtp. For dial command it should look as follows:
It can be requested using answer and streaming_start commands in the same way. The RTP streaming can be sent using media events:
Provided chunks of audio can be in a size of 20 milliseconds to 30 seconds.
The number of bidirectional RTP streams per call is limited to 1.

RTP stream codec

There are the following codecs supported by bidirectional streaming:
  • PCMU, 8 kHz (default)
  • PCMA, 8 kHz
  • G722, 8 kHz
  • OPUS, 8 kHz, 16 kHz
  • AMR-WB, 8 kHz, 16 kHz
  • L16, 16 kHz
When the audio is sent using a different encoding than on the call, it will be transcoded, which may cause a degradation in quality.
The L16 codec provides improved support for AI voice agent integrations by offering reduced latency and eliminating transcoding overhead when interfacing with many AI platforms that natively support linear PCM audio.
The codec can be set when the streaming start is requested. For dial command, it looks as follows:

Sending media files

Media files can also be sent back to the call through the websocket. This is done similarly to the playback_start command when using a base64-encoded mp3 file in the payload. Send a packet to the websocket connection as follows:
The payload, which is a base64-encoded mp3 file, will be played on the call. Multiple media messages will be queued and played in the order they were submitted. Some limitations to be aware of:
  • Media payloads can only be submitted once per second.
  • Media must be base64 encoded mp3.

Clear message

Sending a clear message will immediately stop the media playing on the stream and clear the media queue.

Mark message

Mark messages can be used to keep track of media ending on the stream:
  • You can send a mark message to the stream after a media message.
  • When the media immediately preceding the mark finishes you will receive the same mark back.
Example mark message sent to the stream:
Example mark message received from the stream:
Submitting marks when no audio is played or queued will result in them being immediately sent back. Similarly, the clear message will also result in all queued marks being sent back.

DTMF message

In case of DTMF events on the call, the following message will be sent over websocket:
occurred_at field is a timestamp captured on the Telephony engine side. These are consumed as asynchronous events and in certain circumstances may arrive out of order. This field can be used to ensure proper order.

Error message

In case of any case of error during media streaming, an error frame in the following format is sent:
The list of the potential errors:

Frame numbering

sequence_number and chunk are counted per stream and continue across reconnects.
  • sequence_number counts the frames Telnyx sends. Use it to order frames, not to detect loss: numbers can be skipped, and two frames can share a number. connected and error frames don’t carry a sequence_number.
  • chunk counts media frames, with one counter for both tracks. A gap in chunk means media was lost, but consecutive values don’t prove that none was: media dropped during a reconnect is never numbered.
  • timestamp is the RTP timestamp of the media packet, in the codec’s RTP clock units. It isn’t wall-clock time.

Reconnects

If the WebSocket connection closes unexpectedly, or accepts no data for about 5 seconds, Telnyx reconnects to your stream_url. Reconnects are on by default. To turn them off, set enable_reconnect to false on streaming_start, answer, or dial. After streaming has started, streams that process the call audio before sending it, such as translation streams, don’t reconnect: a disconnect ends them. A reconnect:
  • Uses the same stream_url, including its query string, and the same stream_auth_token. Credentials in the URL must stay valid for the whole call.
  • Is signed the same way as the first connection, with a new telnyx-timestamp. See Verify the WebSocket connection.
  • Starts with a new connected frame, then a new start frame if streaming had already started. sequence_number and chunk continue; they don’t reset.
Currently, Telnyx makes up to 4 connection attempts per disconnect: the first immediately, then after waits of about 0.5, 1, and 2 seconds (±25%). Each attempt can take several seconds, so if all of them fail, the stream typically ends within about 30 seconds of the disconnect. Reconnects are also limited per stream, currently to about 4 in each 30-second window. A disconnect over that limit ends the stream without an attempt. Telnyx doesn’t reconnect when:
  • Your server closes the connection with code 1000, 1007, 1008, or 1011. The stream ends. If streaming had already started, you receive streaming.stopped, not streaming.failed.
  • enable_reconnect is false. The stream ends at the first disconnect. If streaming had already started, you receive streaming.stopped, not streaming.failed.
  • Your server answers a reconnect attempt with a 4xx HTTP status. Telnyx stops retrying, and the stream fails.
While the connection is down:
  • Media and events that arrive during a connection attempt are delivered late if that attempt succeeds. Those that arrive between attempts are dropped. Telnyx doesn’t replay them.
  • No webhook is sent for a reconnect attempt or a successful reconnect.
  • If the connection can’t be restored, you receive streaming.failed with failure_reason set to disconnected. The call continues.

Stream failures

When a stream fails, Telnyx sends the streaming.failed webhook. Its failure_reason field says why: New values may be added, so handle values you don’t recognize.

Example of integration

In the Telnyx GitHub repository there are several examples of integrations with external services using Media streaming: