WebRTC - using Media Stream and PeerConnection API

WebRTC is a great new addition to the HTML5 spec. It gives developers the ability to make apps that can process the audio and video feeds of a user with utmost ease. So instead of handling video drivers, bandwidth, encryption, etc. the developer needs to call simple javascript APIs. It also gives the assurance that the application is going to work on different platform (As usual while all browser vendors are struggling to integrate most of the specification into their browser, Microsoft has come up with it's own implementation).

* The code i have written works on chrome after enabling the flags which are required to make WebRTC work. Also since the WebRTC spec isn't finalised most of the code uses vendor prefixes. 

Creating a P2P audio-video application using WebRTC is a two step process. The first is to ask the user for permissions and get the audio/video feed. The second is streaming and/or processing the feed.

1. Getting the feed

The feed can be obtained by calling getUserMedia (webkitGetUserMedia) of the navigator object.

following is the code which will ask for permission to access audio and video feed (camera and mic) of a user.

navigator.webkitGetUserMedia({audio:true,video:true},onSuccessFunction,onErrorFunction)

The function accepts three parameters. The first is an object containing the list of devices that are required, second argument is the name of the function that will be called on success i.e. when user grants the request and there was no problem while allocating devices and the third argument is the name of the function that gets called when the request fails.

onSuccessFunction(strm){

strm//object containing requested streams

}

2. Processing the feed

Once the stream object is obtained it can be displayed locally in a <video> element, converted to binary data using canvas and then transmitted over websockets for server-side processing , streamed to another machine using peerConnection API.

To display the feed in a video element the stream-object has to be converted to a url.

var url=webkitURL.createObjectURL(strm)

The obtained url can now be assigned to a <Video> element's src attribute.

video1.src=url

video1.play() //stream starts playing

To convert video stream to binary data each frame of the video steam must be displayed in canvas by either using <video> element or using Image() object of javascript and then calling drawimage() on canvas's context.

The binary data can then be obtained by either using toDataUrl() or getImageData() methods of canvas or canvas's context respectively.

Using PeerConnection API :

Peer Connection API lies at the heart of real time communication . It allows P2P (browser-to-browser) streaming of audio and video information. The setup of peer-connection takes place by exchanging some data termed as offer, response and IceCandidates. The object containing the initial request generated by a peer is know as offer, the corresponding answer is know as 'answer'. After exchange of offer and answer startIce method of peer-connection is called and the generated IceCandidates are exchanged (not sure why IceCandidate is necessary) . The following graphic explains the setup for peerconnection.

IMP: While sending offer, answer and candidate the entire object need not (read cannot as i know of no method to send javascript objects) be sent. Instead offer and answer can be converted to SDP by using offer.toSdp() and answer.toSdp(), now since SDP is nothing but a string , it thus can be sent easily after encoding by encodeURIComponent() javascript method. Also it is better to use semicolons to terminate javascript statements .

Peer Connection process:

1. create new peer connection object.
   
   pc1=new webkitPeerConnection00("STUN stun.l.google.com:19302",iceCallback1)
   
   //first argument: stun server, 2nd argument: name of function called when startIce() is called
   //as of now the function name is webkitPeerConnection00, however PeerConnection would be the final name, 00 is for compatibility with the earlier and now depreciated version of peer connection
   
2. add callbacks and audio and/or video stream.
   pc1.onaddstream=gotRemoteStream1 //function to be called when remote stream is obtained
   pc1.addStream(localstream1) //add the stream object obtained from navigator.getUserMedia()
   
3. Repeat the same process for other peer.
4. Create offer object at peer 1 (peer 1=peer that initiates connection, peer 2=the other peer) and set local-description.
   
   var offer=pc1.createOffer(null) //not sure about the argument accepted by this method and why this is null
   pc1.setLocalDescription(pc.SDP_OFFER,offer)
   
5. Send SDP of offer object to peer 2, after URL encoding
   var offer_sdp_encoded=encodeURIComponent(offer.toSdp())
   //send offer_sdp_encoded via AJAX, WebSockets, etc. to peer 1
   
6. Accept offer at peer 2, set remote and local descriptions and send the 'answer' to peer 1
   
   pc2=new webkitPeerConnection00("STUN stun.l.google.com:19302",iceCallback2) pc2.onaddstream=gotRemoteStream2 pc2.setRemoteDescription(pc2.SDP_OFFER,new SessionDescription(decodeURIComponent(offer_sdp_encoded))) //offer_sdp_encoded recieved via AJAX, WebSockets, etc. from peer 1
   
   var answer=pc2.createAnswer(decodeURIComponent(offer_sdp_encoded),{has_audio:true,has_video:true})
   pc2.setLocalDescription(pc2.SDP_ANSWER,answer)
   
   answer_sdp_encoded=encodeURIComponent(answer.tosdp())
   //send answer_sdp_encoded via AJAX, WebSockets, etc. to peer 1
   
7. Call startIce method of pc2 and send the generated candidate and more parameter to peer 1
   
   pc2.startIce()//will cause call iceCallback2 with candidate and more parametersfunction iceCallback2(candidate,more){
   //send candidate.toSdp() after URL encoding and more to peer 1
   }
   
8. Accept answer at peer 1 candidate and more sent from peer 2 (steps 6 and 7), call startIce() and processIceMessage() methods of peerConnection. Send generated IceCandidate and more parameter to peer 2
   
   pc1.setRemoteDescription(pc1.SDP_ANSWER,decodeURIComponent(answer_sdp_encoded))
   pc1.startIce() //will cause call iceCallback1 with candidate and more parameters
   
   function iceCallback1(candidate,more){
   //send candidate.toSdp() after URL encoding and more to peer 1
   }
   
9. call processIceMessage() method of the peerconnection objects (pc1, pc2) of both the peers
   
   pc1.processIceMessage(new IceCandidate(more,candidate));
   pc2.processIceMessage(new IceCandidate(more,candidate));
   
   //This will cause gotRemoteStream2 and gotRemoteStream1 to get executed function gotRemoteStream1(e){ //e.stream contains the audio and/or video stream of peer 2 vid.src=webkitURL.createObjectURL(e.stream) //obtain stream's URL and assign it to a video element } function gotRemoteStream2(e){ //e.stream contains the audio and/or video stream of peer 1 vid.src=webkitURL.createObjectURL(e.stream) //obtain stream's URL and assign it to a video element }
10. NOP

I have created a webpage which can be used to test all the steps written above. It can be found on github .

Resources:

• Google I/O video
• html5rocks.com
• wikipedia: about STUN servers and ICE protocols.
• Peer Connection Test Webpage