Telnyx Video Overview

Getting Started with Video

Requirements

To setup Video all you need is:

• An API Key
• A Client Token to join Rooms you will create

Configuration and Usage

Telnyx Video is enabled using Rooms. A Room represents a virtual place where multiple endpoints using one of Telnyx’s Programmable Video SDKs can connect within a Room Session. Connected users (Room Participants) can share video and audio Tracks in the Room, and receive video and audio Tracks from other Participants in the Room. You can have as many Rooms as you want. For example you could create a long lived Room such as "Daily Standup", or Rooms that you would delete after it's been used like "1-1 with X".

To create a Room you can use the REST API V2 documented here.

A Room can only be joined if the client has generated a Client Join Token which you can create using the REST API V2 documented here. The Client Join Token is short lived and you will be able to refresh it using the Refresh Token provided with it when you request for a Client Join Token.

Once you have a Room and a Client Join Token for it, you can then use our Video SDK on your client side to connect your client to the Room you just created.

Glossary

Term	Description
Room	Resource representing a virtual place where multiple endpoints using one of Telnyx’s Programmable Video SDKs can connect
Room Session	Resource representing a moment where multiple Room Participants were communicating within a given Room
Room Participant	Resource representing an endpoint using one of Telnyx’s Programmable Video SDKs to connect to a given Room
JWT	JSON Web Token. A standard method for representing claims (https://jwt.io/)
Client Join Token (JWT)	A JWT token which contains grants allowing in the Room usecase to join a Room
Refresh Token (JWT)	A JWT token which permits to obtain a new Client Token with same grants
API Key	Secret API Key generated via Portal and used to authenticate Telnyx API calls.
Video SDK	A library used to provide Video features to your application using Telnyx Video platform.

JavaScript Client SDK

The Telnyx Video Client SDK provides all the functionality you need to join and interact with a Telnyx Room from a browser.

Adding Telnyx to your JavaScript client application

Include the @telnyx/video npm module as a dependency:

npm install @telnyx/video@0.1.0-beta.5 --save

Copy Code

Once the package is installed you can import @telnyx/video in your application code.

// main.js
import { Room, createPublisher } from '@telnyx/video';

Copy Code

Now you are ready to connect to a video room that you previously created using the REST API. In order to connect to a room you need a client token that has the necessary grants to join the room. So make sure you generate a client token with the REST API.

const room = new Room(roomId, {
  clientToken: '<CLIENT_TOKEN_FOR_THE_ROOM>',
  publisher: createPublisher({
    context: JSON.stringify({ name: 'Bob', id: 1 }), // send data that you can associate with the publisher participant (for e.g., userId for this participant in your DB)
  }),
});

room.connect().then(() => {
  console.log('You are connected to the room!');
});

Copy Code

The publisher is a type of participant, who can publish media in a room. You can think of them as the local participant of the client.

The room object has an internal state that you can use to rerender your application UI. In order to get the latest state we should first setup a callback that will be invoked whenever the state changes.

const stateCallback = (state) => {
  // the state object is immutable and can be easily integrated with most modern UI libraries like React, Vue etc. We will explore the structure of this object later.
};

room.on('state_changed', stateCallback);

Copy Code

Understanding the state of the room

The state_changed event callback contains the state of the SDK at that point in time. This is an immutable object that you can use in most modern UI libraries like React and Vue to rerender your UI components.

The Typescript definition of the State is helpful in understanding the structure of this object.

type Status =
  | 'initialized'
  | 'connecting'
  | 'connected'
  | 'disconnecting'
  | 'disconnected';

interface State {
  status: Status;
  participants: {
    [id: string]: Participant;
  };
  streams: {
    [id: string]: Stream;
  };
  publisher: Publisher;
  subscriptions: {
    [participantId: string]: {
      [key: string]: Subscription;
    };
  };
}

Copy Code

Whenever the state of the SDK changes (for e.g., a new participant joined) the state_changed callback is invoked with a new immutable object that represents the current state of the SDK. Since most modern UI libraries are able to compare the current and previous immutable states and render only the components that changed, it's easier to integrate the SDK with them rather than depending on multiple event callbacks.

For e.g., when we created the room object in the example, the initial state of the SDK would result in an object similar to the on shown below. Note that the ID of the local participant is unique and are generated when you create the local participant.

{
  status: "initialized",
  participants: {
    "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa": {
      id: "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa",
      context: '{"id":312642,"username":"Bob"}',
      streams: {},
      isVirtual: false,
      isRemote: false,
    },
  },
  streams: {},
  publisher: {
    participantId: "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa",
    streamsPublished: {},
  },
  subscriptions: {},
}

Copy Code

When we call the room.connect() method in the example the state of the SDK will change to the following ...

{
  status: "connecting",
  // the rest of the object will remain the same as only the status will change
}

Copy Code

This new state will be available to your application via the state_changed callback. Since state is an immutable object the only difference between the initial state and the new state is the status property. If you are using a modern UI library like React you can easily rerender the components to show that the application is connecting to the room.

Once connected the state changes to ...

{
  status: "connected",
  // the rest of the object will remain the same as only the status will change
}

Copy Code

Publishing your first stream

Now that we are connected we can publish streams to this room. A stream can contain one audio track and one video track. We will use the getUserMedia API to get the tracks.

const constraints = { audio: true, video: true };

navigator.mediaDevices.getUserMedia(constraints).then((stream) => {
  // the stream is an instance of MediaStream (https://developer.mozilla.org/en-US/docs/Web/API/MediaStream)
  room.publish('self', {
    audioTrack: stream.getAudioTracks()[0], // get the first audio track
    videoTrack: stream.getVideoTracks()[0], // get the first video track
  });
});

Copy Code

The room object provides a publish method that you can use to publish the tracks that you received from the getUserMedia call. The first argument to publish is a string that acts as the key you can use to refer to this specific stream. You can use any valid string for the key as long as it's consistent in your application. For e.g. here we're using 'self' for the camera/mic stream but you can use 'presentation' as the key when you publish the video track of the screen. Using different keys allows you to create logical separation between types of streams in your application.

When the publish method is called you will receive a new state to the state_changed event callback with the newly created stream.

{
  status: "connected",
  participants: {
    "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa": {
      id: "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa",
      context: '{"id":312642,"username":"Bob"}',
      streams: {
        self: {
          streamId: "bdc0140f-6452-4791-9b82-7f4f1201ccc2",
        },
      },
      isVirtual: false,
      isRemote: false,
    },
  },
  streams: {
    "bdc0140f-6452-4791-9b82-7f4f1201ccc2": {
      id: "bdc0140f-6452-4791-9b82-7f4f1201ccc2",
      audioTrack: MediaStreamTrack, // the audio track that was provided to the `publish` method
      videoTrack: MediaStreamTrack, // the video track that was provided to the `publish` method
      audioEnabled: true,
      videoEnabled: true,
      isSpeaking: false,
      audioStatus: "inactive",
      videoStatus: "inactive",
    },
  },
  publisher: {
    participantId: "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa",
    streamsPublished: {
      self: {
        status: "pending", // the stream "self" is pending publication
      },
    },
  },
  subscriptions: {},
}

Copy Code

At this point the stream is pending publication as you can see from the publisher object, which updates you on the state of the streams you are publishing.

For simplicity you can rely on the status of the stream key within publisher.streamsPublished object to know the status of the publication.

Since publishing the track will involve negotiating a WebRTC connection a generated SDP is sent to the Telnyx media server, which results in the following state. The information from the generated SDP is used to update the state with the codec that will be used as well as the status.

Note that you can't publish another stream of the same key ("self" in our case) until this stream has finished publishing.

{
  status: "connected",
  participants: {
    "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa": {
      id: "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa",
      context: '{"id":312642,"username":"Bob"}',
      streams: {
        self: {
          streamId: "bdc0140f-6452-4791-9b82-7f4f1201ccc2",
        },
      },
      isVirtual: false,
      isRemote: false,
    },
  },
  streams: {
    "bdc0140f-6452-4791-9b82-7f4f1201ccc2": {
      id: "bdc0140f-6452-4791-9b82-7f4f1201ccc2",
      audioTrack: MediaStreamTrack,
      videoTrack: MediaStreamTrack,
      audioCodec: "opus", // audio will use opus
      videoCodec: "VP8", // video will use VP8
      audioEnabled: true,
      videoEnabled: true,
      isSpeaking: false,
      audioStatus: "sending", // audio will be sent once the WebRTC connection is established
      videoStatus: "sending", // video will be sent once the WebRTC connection is established
    },
  },
  publisher: {
    participantId: "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa",
    streamsPublished: {
      self: {
        status: "pending",
      },
    },
  },
  subscriptions: {},
}

Copy Code

The stream object is the most complex object in the state of the SDK. However you don't have to worry about all of these properties at this point.

Once the WebRTC connection is negotiated and the stream is successfully published the publisher state would look like this

  publisher: {
    participantId: "e54c8eed-f84c-433f-a024-0a532650d511",
    streamsPublished: {
      self: {
        status: "published", // the stream "self" is now published
      },
    },
  },

Copy Code

At this point the stream is available to be subscribed to by all other clients connected to the same room.

Knowing when a new participant joins the room

A Telnyx room can have multiple participants and you can get a list of all the participants connected to the room by using the participants property in the state.

If the participant is not publishing any streams the streams property of that participant will be an empty object.

Let's say that another participant joins the same room from a different browser session with the following context

const room = new Room(roomId, {
  clientToken: '<CLIENT_TOKEN_FOR_THE_ROOM>',
  publisher: createPublisher({
    context: JSON.stringify({ name: 'Alice', id: 2 }),
  }),
});

Copy Code

Since Bob is already publishing a stream as a participant in the room, when Alice connects the state will look something like this...

{
  status: "connected",
  participants: {
    "07bd6f20-23e5-45df-b289-f350fc553c14": {
      id: "07bd6f20-23e5-45df-b289-f350fc553c14",
      context: '{"id":490503,"username":"Alice"}',
      streams: {},
      isVirtual: false,
      isRemote: false,
    },
    "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa": {
      id: "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa",
      context: '{"id":312642,"username":"Bob"}',
      streams: {
        self: {
          streamId: "bdc0140f-6452-4791-9b82-7f4f1201ccc2",
        },
      },
      isVirtual: false,
      isRemote: true,
    },
  },
  streams: {
    "bdc0140f-6452-4791-9b82-7f4f1201ccc2": {
      id: "bdc0140f-6452-4791-9b82-7f4f1201ccc2",
      audioCodec: "opus",
      videoCodec: "VP8",
      audioEnabled: true, // audio is enabled by the publisher (Bob) of this stream
      videoEnabled: true, // video is enabled by the publisher (Bob) of this stream
      isSpeaking: false,
      audioStatus: "inactive", // Alice is not receiving the audio
      videoStatus: "inactive", // Alice is not receiving the video
    },
  },
  publisher: {
    participantId: "07bd6f20-23e5-45df-b289-f350fc553c14", // this is Alice!
    streamsPublished: {},
  },
  subscriptions: {},
}

Copy Code

You can see that there are two participants in the room and Bob is publishing a stream with the key self. You might have also noticed that the participant object for Bob has isRemote set to be true. All remote participants will have this flag set to be true.

At this point we know that Bob is publishing a stream with the key self but Alice has not subscribed to this stream yet. With the SDK you have control over the streams that you want to subscribe to control the bandwidth usage on the client.

Subscribing to a remote stream

Subscribing to a remote stream is quite easy to do with the subscribe method available in the room instance.

room.subscribe('8f20b6a7-0556-41fc-9e2a-3a3ad81377fa', 'self', {
  audio: true, // receive audio
  video: true, // receive video
});

Copy Code

This will start the WebRTC negotiation to receive the remote stream and the subscriptions state will change to something like ...

  subscriptions: {
    "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa": {
      self: {
        audio: true,
        video: true,
        status: "pending",
      },
    },
  },

Copy Code

Subscriptions are organised by participant ID and the stream key. Similar to when you publish a track, you can use the status to know whether the subscription has started or not.

Once the WebRTC negotiation is completed and the stream is flowing, the state will look something like this ...

  // only showing the relevant parts of the state for clarity
  streams: {
    "bdc0140f-6452-4791-9b82-7f4f1201ccc2: {
      id: "bdc0140f-6452-4791-9b82-7f4f1201ccc2,
      audioTrack: MediaStreamTrack, // the remote audio track
      videoTrack: MediaStreamTrack, // the remote video track
      audioCodec: "opus",
      videoCodec: "VP8",
      audioEnabled: true,
      videoEnabled: true,
      isSpeaking: false, // if the user is speaking this will be true
      audioStatus: "receiving", // audio track data is being received
      videoStatus: "receiving", // video track data is being received
    },
  },
  subscriptions: {
    "8f20b6a7-0556-41fc-9e2a-3a3ad81377fa": {
      self: {
        audio: true,
        video: true,
        status: "started", // the subscription has started
      },
    },
  },

Copy Code

If the remote participant unpublishes this stream the SDK will automatically unsubscibe from the stream and the stream will disappear from the state.

Knowing when a participant is speaking

You can use the isSpeaking property in the stream object to know if the audio levels of that stream is high enough to be considered as speaking. Please note that this approach might change in the future.

Leaving the room

When the participant decides to leave the room you can use the disconnect method in the room instance to leave the room.

room.disconnect();

Copy Code

This will update the status property in the state to disconnecting and then to disconnected once the participant has left the room. On the remote side the participant and its associated streams will be unsubscribed (if already subscribed) and removed from the state.