Telnyx Other APIs: Video — Full Documentation
Complete page content for Video (Other APIs section) of the Telnyx developer docs (https://developers.telnyx.com). Root index: https://developers.telnyx.com/llms.txt · Lightweight index for this subsection: https://developers.telnyx.com/development/llms/other-apis-video-llms-txt.md
Getting Started
Source: https://developers.telnyx.com/docs/video/get-started.mdTelnyx Video Rooms are enabled using the Rooms API. You can create as many rooms as needed using this API. To access a Telnyx Video Room, a Client Join Token will first need to be generated in JWT form. You can use our HTTPS API and authenticate using the API Key associated with your Mission Control account under API Keys. Find out more about authenticating with API V2 here. Adding Telnyx Rooms functionality to your app can be done using our JS SDK (mobile SDKs coming soon). To gain access to a live Video Rooms sample app, reach out to our sales team today! Check out the full API reference today!
Glossary
Room Resource representing a virtual place where multiple endpoints using one of Telnyx’s Programmable Video SDKs can connect. Room Session Resource representing a moment where multiple Room Participants were communicating within a given Room. Room Participant Resource representing an endpoint using one of Telnyx’s Programmable Video SDKs to connect to a given Room. JWT JSON Web Token. A standard method for representing claims. Client Join Token (JWT) A JWT token which contains grants allowing in the Room usecase to join a Room. Refresh Token (JWT) A JWT token which permits to obtain a new Client Token with same grants. API Key Secret API Key generated via Portal and used to authenticate Telnyx API calls. Video SDK A library used to provide Video features to your application using Telnyx Video platform. Configuration and usage Telnyx Video is enabled using Video Rooms. A Video Room represents communications session among multiple endpoints using one of Telnyx’s Programmable Video SDKs. Connected users (Participants) can share video and audio Tracks with the Room, and receive video and audio Tracks from other Participants in the Room. You can as many Rooms as you want. For example you could create a long lived Room such as “Daily Standup”, or Rooms that you would delete after it’s been used like “1-1 with X”. To create a Video Room you can use the REST API V2 documented{” ”} here. A Video Room can only be joined if the client owns a Client Join Token, you can create it using the REST API V2 documented{” ”} here. The Client Join Token is short lived and you will be able to refresh it using the{” ”} Refresh Token provided with it when you request for a Client Join Token. Once you have a Video Room and an Client Join Token for it, you can then use our Video SDK on your client side to connect your client to the Room you just created.Concepts
Architecture
Video Rooms is a platform that enables developers to add audio and video capability to Web, Android, and iOS applications.
The platform consists of REST APIs, Client SDKs, and our mission control portal that makes it really easily to capture, stream, record, and render live audio and video.
A video application built with Video Rooms has to parts:
- Client: Our client side Javascript, iOS, and Android SDKs used to interact with a
Roominstance - Server: Our REST APIs and portal to create/manage room and session, configuring recording, or leverage our
Participants APIto moderator participants in aRoom.
Terms
Understanding the basic concepts of the video SDK will help you understand how it works. These concepts apply in general across all of our platforms.-
A
Roomrepresents a real time audio/video/screen share session with other people or participants. It is fundamental to building a video application. -
Room Statetracks the state of the room as it changes making it extremely easy to understand what’s happened to aRoom.For example: Room State could change due to a Local Participant has started publishing a stream or because a Remote Participant left. A Stream represents the audio/video media streams that are shared by Participants in a Room
-
A
Participantrepresents a person inside aRoom. EachRoomhas oneLocal Participantand one or moreRemote Participants. -
A
Streamrepresents the audio/video media streams that are shared byParticipantsin aRoom- A
Streamis indentified by it’sparticipantIdandstreamKey
- A
-
A
Subscriptionis used to subscribe to aStreambelonging to aRemote Participant
Dive a bit deeper
Dive a little bit deeper into ourVideo Rooms platform to get a better understand of what its capable of, and what you can buid. Learn more about our Client SDKs and Server APIs.
Client SDKs
Our Javascript SDK API reference which details the API of our SDK including behaviors of theRoom class and the Events that triggers and how they function.
Server APIs
- Rooms - manage Rooms
- Client Access Tokens - manage client access tokens needed to interact with a
Room - Sessions - manage room sessions, end a session, and mute/unmute/kick all participants in a given session.
- Participants - search for participants based on a number of filters like
session id - Recordings - manage recordings, including bulk delete.
- Compositions - create and manage compositions.
SDKs
Getting Started
Source: https://developers.telnyx.com/docs/video/javascript-sdk.mdThe Telnyx Video Client SDK provides all the functionality you need to join and interact with a video room from a browser.
Adding Telnyx to your JavaScript client application
Include the@telnyx/video npm module as a dependency:
npm install @telnyx/video --save
After pasting the above content, Kindly check and remove any new line added
Then, import @telnyx/video in your application code.
Understanding the state of the video room
Thestate_changed event callback contains the state of the SDK at that point in time. This is an immutable object that you can use in most modern UI libraries like React and Vue.
The Typescript definition of the State is helpful in understanding the structure of this object.
state_changed callback is invoked with a new immutable state that represents the current state of the SDK. Since most modern UI libraries are able to compare the two immutable states and render only the components that changed it makes it easier to integrate the SDK with them rather than depending on multiple event callbacks.
For e.g., based on how we initialized it in the above code example the initial state of the SDK would result in an object give below. Note that the ID of the local participant is unique and are generated based on UUID v4 standard when you create the local participant.
room.connect() method in the example code the state of the SDK will change to the following …
state_changed callback. Since state is immutable the only difference between the initial state and the new state is the status property. If you are using a modern UI library like React you can easily rerender the components to show that the application is connecting to the room.
When successfully connected the state changes to …
Publishing your local camera and mic stream
In order to publish a stream you need to define the constraints of the media. The simplest form of the constraints is …publish method available to you via the room object.
publish is a string that acts as the key that you can use to refer to this specific stream. You can use any valid string for this as long as you are consistent in your application. For e.g. here we’re using ‘self’ for the camera/mic stream but you could use ‘presentation’ as the key for when you publish video of the screen. We will get to how you can publish your screen in a short while.
When you make the request to publish a stream, the browser will ask for the necessary permissions required to access the camera and mic. Once the permissions are acquired the SDK will configure and publish the stream to the room you are connected to.
At this point you will receive the new state to the state_changed event callback with the newly created stream.
isPublishing property of the stream is true now. This means that the stream is being published, which involves creating the SDP (Session Description Protocol) and establishing the WebRTC connection. You can’t publish another stream of the same key (“self” in our case) until this stream is published or removed (by unpublishing).
Once the SDK acquires the media tracks from the camera and mic of the local participant a new state will be returned using the state_changed callback. This state will contain the audio and video track from the local media devices.
The streams object at this point will look like this
source or the media tracks individually from audioTrack and videoTrack to show the media in your UI.
You can also see that the audioActive and videoActive flags are now true indicating the stream is publishing audio and video respectively.
Another property that was updated in the new state is the isConfiguring flag. You can ignore this in most use-cases as this indicates if the WebRTC connection is being negotiated.
Once the WebRTC connection is negotiated and the stream is successfully published the stream state would look like this
Knowing when a new participant joins the room
A video room can have multiple participants and you can get a list of all the participants connected to the room at the time by using theparticipants property in the state.
If the participant is not publishing any streams the streams property of that participant will be an empty object.
Let’s imagine that another participant joins the same room from a different browser session with the following context
participants object that maps to the remote particpant from the previous browser session. This remote participant also has a stream with the key “self”.
The stream object for “self” has the structure very similar to the one we saw before but has a few new properties.
The ones that are particularly interesting are subscription, videoCodec, and audioCodec. The properties for video and audio codec can be used to decide if the browser has the capabilities to decode the video and audio tracks.
Then there is subscription.status, which is unpublished at the moment.
Subscribing to a remote stream
At this point we are ready to subscribe to the remote stream published by the participant. In order to do that we use thesubscribe method from the room object.
subscription.status will change to subscribed.
You can use the source property, which is an instance of MediaStream to render the media on the browser.
If the remote participant unpublishes this stream the SDK will automatically unsubscibe from the stream and the stream will disappear from the state.
Room and Events
Source: https://developers.telnyx.com/docs/video/javascript-sdk/room-events.mdTelnyx’s video JavaScript SDK lets you manage room events and join/leave notifications. We’ve made it much easier to add video capability to your web application using our Javascript SDK. The
Room object and the Events that occur in a Room is the API you will use to build your video application. We’ve extensively documented the interface for the Room and Events objects, below.
Room
Room events
Tutorial
Source: https://developers.telnyx.com/docs/video/javascript-sdk/javascript-video-tutorial.md
In 10 minutes we’ll build
A simple web app in vanilla javascript to make a video with audio from a caller to a callee.Get an API Key
You need a Telnx account to create an API key so you can interact with our Rooms API
- If you don’t have one please: Sign up for a free account
- Navigate to API Keys section and create an API Key by clicking
Create API Keybutton. - Copy your API key
Create a Room
Let’s use our Rooms API to create a room. Request Don’t forget to updateYOUR_API_KEY here.
id field, this is the id for your newly created Room.
Generate a client token
You’ll need a client access token to join the room. To create one use our REST APIrooms/create endpoint.
You’ll need to replace ROOM_ID in in the url of the command with your room id, above. As well as YOUR_API_KEY as you have in previous steps.
Request
token field in the response is the client access token you’ll use to join the room.
🏁 Checkpoint - run the app
At this point you should have the following:- An API Key
- A room ID
- A room client access token
Let’s write some code
Basic concepts of the video SDK Here are some basic concepts of the SDK that will help you better understand how it works.-
A
Roomrepresents a real time audio/video/screen share session with other people or participants. It is fundamental to building a video application. -
A
Participantrepresents a person inside aRoom. EachRoomhas oneLocal Participantand one or moreRemote Participants. -
A
Streamrepresents the audio/video media streams that are shared byParticipantsin aRoom- A
Streamis indentified by it’sparticipantIdandstreamKey
- A
-
A
Participantcan have one or moreStream’s associated with it. -
A
Subscriptionis used to subscribe to aStreambelonging to aRemote Participant
Room instance that we’ll need to finish building this app that makes a video call.
connected- triggers when a room instance has connected to the serverparticipant_joined- triggers when a remote participant joins the roomstream_published- triggers when a stream has started being published to the roomsubscription_started- triggers when subscription to remote stream has started
🏁 Checkpoint
It might be helpful to take a look at the full list ofEventsavailable on aRoominstance.
Connect to the room and get local media
NOTE: We can start implementing our code after the room.initialize block inside the sandbox above.
Let’s connect to the room and use the standard WebRTC API to get the audio and video tracks from the device.
Publish the stream and render it
We want to a add/publish a stream the room which consists of the audio and video track we received from the user’s device, above. We’ll give the stream a key of “caller”, which is how the stream is identified.room.connected block looks like this:
Subscribe to the callee’s remote stream
As a caller, how do we know when the callee has joined the room or has started publishing a stream? That’s where theparticipant_joined and the stream_published events that we went over earlier, come into play.
We need to listen for teh stream_published event like this:
stream_published event is triggered for both local and remote streams. __
In our case, we want to know when the callee’s remote stream starts publishing so we can subscribe to it. We already know that the caller has published a “caller” stream so we can ignore that event.
Subscribing to a stream
When a stream is published in a room it doesn’t mean it’s audio and video tracks are accessible, yet. In order to access a remote stream’s track we must explicitly subscribe to that stream and wait for the subscription_started event.
Render the callee’s remote stream
We’re almost there! Now, we can listen to thesubscription_started event and use args from that handler to get the callee’s remote stream and render it to the DOM.
Run the app
To run the app:- Open the sandbox, the browser here represents the caller’s app
- Open the app in another tab, this represents the callee’s application which you’ll need to interact with.
- Click the call button
- You should be prompted by the browser for camera/micrphone access please allow.
- You should also see video from your device’s camera and hear audio from the microphone
- You should see some console log, if things are working property
- If you see errors your client token has expired and you need to regenerate it.
- Click the call button
- Notice in the caller’s app a stream subscription was logged
- The callee’s video/audio stream is rendered.
Android SDK
Source: https://developers.telnyx.com/docs/video/android-client-sdk.mdThe Telnyx Video Android SDK is simple to use and makes it easy to get started with video calling. With this SDK, you can easily add video calling to your app with just a few lines of code. It provides all the functionality you need to join and interact with a Telnyx Room from an Android application.
A link to the repo can be found here:
Android SDKProject structure
- SDK project: sdk module, containing all Telnyx SDK components as well as tests.
- Demo application: app module, containing a sample demo application utilizing the sdk module.
Adding the SDK to your Android client application
Add Jitpack.io as a repository within your root level build file:Before connecting to a Room
Get an API Key
You’ll need an API key which is associated with your Mission Control Portal account under API Keys. You can learn how to do that here. An API key is your credential to access our API. It allows you to:- to authenicate to the Rest API
- to manage your access tokens
Create a Room to join (if it doesn’t exist)
In order to join a room you must create it, if it doesn’t already exist. See our Room Rest API to create one. There’s also additional resources on other endpoints available to perform basic operations on aRoom.
Generate an a client token to join a room
In order to join a room you must have a client token for thatRoom. The client token is short lived and you will be able to refresh it using the refresh token provided with it when you request a client token.
Please see the docs here to learn how to create a client token.
Now you are ready to connect to a video room that you previously created using the REST API.
Connect to Room
To connect, you’ll need to provide a participantName that will identify your user in that room. You’ll also need to provide an instance of ExternalData that will contain a username of type String and an Integer id. You will also provide your Android application’s context in contextPublish video/audio stream
To publish as video or audio stream, we will need an instance of PublishConfigHelper with the application context, camera direction, streamKey (unique for each stream published), and streamId (unique for each stream published)Remove video/audio track
To remove a video or audio track, publishConfigHelper has to be modified in order to remove the unwanted trackRemove stream
By removing the stream, we will remove all tracks added to it.Video/Audio Observables
The Telnyx Video SDK for android works with mutable live data, and all the information you need to build your UI is provided through observables that will contain the most up to date information of the state of the room, such as current participant list, talking events, stream information, participants added, participants leaving, etcState Observable
val status: String
is the status of the Room session, when that action happened
val participants: List
is the list of participants present in a room. A Participant is a UI representation in variables, for an attendee to the room session.
val streams: HashMap
this hash map, contains a track of the currently available streams and the id of the publisher streaming them.
A stream, will contain tracks for video, audio or both.
val publishers: List
This will track all publishers in the room. A Publisher is an instance of an attendee or participant sharing some stream content in the room.
A single Participant, sharing multiple streams can be also linked to multiple Publisher ids.
val subscriptions: List
Each time a publisher starts streaming information, we will have the option of subscribe/unsubscribe to/from it. This list will track all the subscriptions.
Event observables
Some observables will have a MutableLiveData of Event. Event is a wrapper of LiveData, and provide the means to ensure we only handle an observable once, no matter how many times we’re set to observe it. If the contents have already been handled, we won’t get that content again, unless we peekContent() instead.Participants Observable
Joined Room Observable
Joined Participant Observable
Leaving participant id Observable
Connected to room Observable
Participant stream changed Observable
Stream status
Subscribe to a video stream
In order to subscribe to a video stream, there are 3 actions that needs to be performed:Remove subscription to a *video stream
To remove a subscription we need to issueParticipant Talking Observable
Stats
We provide the method getWebRTCStatsForStream() to retrieve WebRTC stats This request brings stats one time only, so if you want to keep receiving stats, you will need to use some sort of runnable or coroutine to recursively request for them such as:Remote Video stats
WEBRTC’s RTCStatsReport can be parsed and mapped to RemoteVideoStreamStatsLocal Video stats
WEBRTC’s RTCStatsReport can be parsed and mapped to LocalVideoStreamStatsRemote audio stats
WEBRTC’s RTCStatsReport can be parsed and mapped to RemoteAudioStreamStatsLocal audio stats
WEBRTC’s RTCStatsReport can be parsed and mapped to LocalAudioStreamStatsiOS SDK
Source: https://developers.telnyx.com/docs/video/ios-client-sdk.mdThe Telnyx Video iOS SDK provides the functionality you need to join and interact with a video room from an iOS application.
A link to the repo can be found here:
iOS SDKIf you prefer to jump right in
Have a look at our demo app: Telnyx Meet.An overview of the Video API
These are important concepts to understand.-
A
Roomrepresents a real time audio/video/screen share session with other people or participants. It is fundamental to building a video application. -
A
Participantrepresents a person inside aRoom. EachRoomhas oneLocal Participantand one or moreRemote Participants. -
Room Statetracks the state of the room as it changes making it extremely easy to understand what’s happened to aRoom.Room Statecould change due to aLocal Participanthas started publishing a stream or because aRemote Participantleft.
-
A
Streamrepresents the audio/video media streams that are shared byParticipantsin aRoom- A
Streamis indentified by it’sparticipantIdandstreamKey
- A
-
A
Participantcan have one or moreStream’s associated with it. -
A
Subscriptionis used to subscribe to aStreambelonging to aRemote Participant
API of a Room
This should give you high level overview of the Room API and it’s functionality.Events that are triggered in a Room
Here’s a list of events that will fire as you make API calls.Understanding the state of the Room
Everything in the SDK centers around theRoom object. When the state of the Room changes (e.g. a new participant joins or a remote participant starts publishing a stream) the onStateChanged event is triggered.
The event is invoked with a state parameter which contains the current of the state of the Room.
Before getting started
Install the SDK
Currently, the Telnyx iOS Video SDK can be installed using CocoaPods. For instructions on that check out our releases repo for iOSGet an API Key
You’ll need an API key which is associated with your Mission Control Portal account under API Keys. You can learn how to do that here. An API key is your credential to access our API. It allows you to:- to authenicate to the Rest API
- to manage your access tokens
Create a Room to join (if it doesn’t exist)
In order to join a room you must create it, if it doesn’t already exist. See our Room Rest API to create one. There’s also additional resources on other endpoints available to perform basic operations on aRoom.
Generate an a client token to join a room
In order to join a room you must have a client token for thatRoom. The client token is short lived and you will be able to refresh it using the refresh token provided with it when you request a client token.
Please see the docs here to learn how to create a client token.
Code Examples
Enough already let’s get to the code. Here are some code examples to get your wet feet on how to start building something with the iOS video SDK.Participating in a Room
Connect to a room
First, you’ll need to create aRoom instance and then connect to it. Once you’re connected to a room, you can start sharing audio/video streams with other participant in the rooms.
Important Note:
This simply creates an instance of a Room in code it does not use the Rooms Rest API to create a room, mentioned above in “Create a Room to join”*
Room only has one Local Participant but can have multiple Remote Participants.
What is Room context?
Context is any details you want to include about theLocalParticipant of the Room.
For instance, let’s say you want to use context to identify a Participant with fields from an external system. You could pass a userId and username as context, like the code snippet above.
These details will be available to all RemoteParticipants in the Room, when they are notified about your presence in the Room.
Working with local media
Publishing audio and/or video from your camera or microphone works by usingMediaDevices. MediaDevices is a helper class that we provide for you to make it easy to grab local media from your device.
Setting the quality of the local video
You can set the camera resolution and fps usingMediaDevices.
Publishing a stream
Once you have tracks from say, a local media device like video from a camera and/our audio from your micrphone you can use those to create aStream and publish it in the Room.
Unpublishing a stream
If you can longer want to continue publishing a stream you canunpublish it.
Naturally the stream you want to unpublish must be added already.
Working with Remote Participants and Streams
A remote participant who joins or leaves the room
When a remote participant joins a a room you will be notified with theRoom.onParticipantJoined event. And similiarly with Room.onParticipantLeaving when a remote participant leaves.
You can use these events to keep track of participants in the room.
Remote participants already in the room
When you connect to aRoom there may already be remote participants in the Room. To understand who is in the Room after you connect use the onParticipantJoined event.
Display a remote participant’s media
In order to understand how to display a remote participants’ media let’s review on subscriptions work in the API. It might be helpful to review the Video API overview to get a better understanding of how aRoom is modeled, especially Stream and Subscription.
Major 🔑 about Subscriptions
In order to display media from a remote stream you need to subscribe to it.
It’s important to understand that your Room doesn’t automatically subscribe to a remote stream being published. It’s your choice to decide whether to subscribe to a given stream.
Subscribing to a stream
So let’s say the app your building has two users - let’s them call them Alice and Bob. First, Alice joins theRoom and starts publishing a stream with audio from her microhone and video from her camera like this:
NOTE: The app on Alice’s device runs the following code…
Handling remote streams that are already publishing in the Room
After you connect to a Room there may be remote participants already in theRoom who are publishing streams in the Room.
To deal with that use the onStreamPublished event:
Disconnecting from a Room
To disconect from a room do:Room all Remote Participant’s will be notified that you’ve left the Room because the Room.onParticipantLeft event will fire on their Room instance.
API Reference (Video)
Room Compositions
- View a list of room compositions.
- Create a room composition.: Asynchronously create a room composition.
- View a room composition.
- Delete a room composition.: Synchronously delete a room composition.
Room Participants
Room Recordings
- View a list of room recordings.
- Delete several room recordings in a bulk.
- View a room recording.
- Delete a room recording.: Synchronously delete a Room Recording.
Room Sessions
- View a list of room sessions.
- View a room session.
- End a room session.: Note: this will also kick all participants currently present in the room
- Kick participants from a room session.
- Mute participants in room session.
- Unmute participants in room session.
- View a list of room participants.
Rooms
- View a list of rooms.
- Create a room.: Synchronously create a Room.
- View a room.
- Update a room.: Synchronously update a Room.
- Delete a room.: Synchronously delete a Room. Participants from that room will be kicked out, they won’t be able to join that room anymore, and you won’t be charged anymore for…
- View a list of room sessions.
Rooms Client Tokens
- Create Client Token to join a room.: Synchronously create an Client Token to join a Room. Client Token is necessary to join a Telnyx Room. Client Token will expire after
token_ttl_secs, a Refres… - Refresh Client Token to join a room.: Synchronously refresh an Client Token to join a Room. Client Token is necessary to join a Telnyx Room. Client Token will expire after
token_ttl_secs.