Friday 28 September 2012

WebRTC - using Media Stream and PeerConnection API

WebRTC is a great new addition to the HTML5 spec. It gives developers the ability to make apps that can process the audio and video feeds of a user with utmost ease. So instead of handling video drivers, bandwidth, encryption, etc. the developer needs to call simple javascript APIs. It also gives the assurance that the application is going to work on different platform (As usual while all browser vendors are struggling to integrate most of the specification into their browser, Microsoft has come up with it's own implementation).

The code i have written works on chrome after enabling the flags which are required to make WebRTC work. Also since the WebRTC spec isn't finalised most of the code uses vendor prefixes. 

Creating a P2P audio-video application using WebRTC is a two step process. The first is to ask the user for permissions and get the audio/video feed. The second is streaming and/or processing the feed.

  1. Getting the feed

The feed can be obtained by calling getUserMedia (webkitGetUserMedia) of the navigator object.
following is the code which will ask for permission to access audio and video feed (camera and mic) of a user.

navigator.webkitGetUserMedia({audio:true,video:true},onSuccessFunction,onErrorFunction)

The function accepts three parameters. The first is an object containing the list of devices that are required, second argument is the name of the function that will be called on success i.e. when user grants the request and there was no problem while allocating devices and the third argument is the name of the function that gets called when the request fails.

onSuccessFunction(strm){
strm//object containing requested streams
}


  1. Processing the feed
Once the stream object is obtained it can be displayed locally in a <video> element, converted to binary data using canvas and then transmitted over websockets for server-side processing , streamed to another machine using peerConnection API.

To display the feed in a video element the stream-object has to be converted to a url.

var url=webkitURL.createObjectURL(strm)

The obtained url can now be assigned to a <Video> element's src attribute.

video1.src=url
video1.play() //stream starts playing

To convert video stream to binary data each frame of the video steam must be displayed in canvas by either using <video> element or using Image() object of javascript and then calling drawimage() on canvas's context.
The binary data can then be obtained by either using toDataUrl() or getImageData() methods of canvas or canvas's context respectively.

Using PeerConnection API :

Peer Connection API lies at the heart of real time communication . It allows P2P (browser-to-browser) streaming of audio and video information. The setup of peer-connection takes place by exchanging some data termed as offer, response and IceCandidates. The object containing the initial request generated by a peer is know as offer, the corresponding answer is know as 'answer'. After exchange of offer and answer startIce method of peer-connection is called and the generated IceCandidates are exchanged (not sure why IceCandidate is necessary) . The following graphic explains the setup for peerconnection.



IMP: While sending offer, answer and candidate the entire object need not (read cannot as i know of no method to send javascript objects) be sent. Instead offer and answer can be converted to SDP by using offer.toSdp() and answer.toSdp(), now since SDP is nothing but a string , it thus can be sent easily after encoding by encodeURIComponent() javascript method. Also it is better to use semicolons to terminate javascript statements .

Peer Connection process:

  1. create new peer connection object.

    pc1=new webkitPeerConnection00("STUN stun.l.google.com:19302",iceCallback1)

    //first argument: stun server, 2nd argument: name of function called when startIce() is called
    //as of now the function name is webkitPeerConnection00, however PeerConnection would be the final name, 00 is for compatibility with the earlier and now depreciated version of peer connection
  2. add callbacks and audio and/or video stream.
    pc1.onaddstream=gotRemoteStream1 //function to be called when remote stream is obtained
    pc1.addStream(localstream1) //add the stream object obtained from navigator.getUserMedia()
  3. Repeat the same process for other peer.
  4. Create offer object at peer 1 (peer 1=peer that initiates connection, peer 2=the other peer) and set local-description.

    var offer=pc1.createOffer(null) //not sure about the argument accepted by this method and why this is null
    pc1.setLocalDescription(pc.SDP_OFFER,offer)
  5. Send SDP of offer object to peer 2, after URL encoding
    var offer_sdp_encoded=encodeURIComponent(offer.toSdp())
    //send 
    offer_sdp_encoded via AJAX, WebSockets, etc. to peer 1
  6. Accept offer at peer 2, set remote and local descriptions and send the 'answer' to peer 1

    pc2=new webkitPeerConnection00("STUN stun.l.google.com:19302",iceCallback2) pc2.onaddstream=gotRemoteStream2 pc2.setRemoteDescription(pc2.SDP_OFFER,new SessionDescription(decodeURIComponent(offer_sdp_encoded)))
    //offer_sdp_encoded recieved via AJAX, WebSockets, etc. from peer 1

    var answer=pc2.createAnswer(decodeURIComponent(offer_sdp_encoded),{has_audio:true,has_video:true})
    pc2.setLocalDescription(pc2.SDP_ANSWER,answer)

    answer_sdp_encoded=encodeURIComponent(answer.tosdp())
    //send 
    answer_sdp_encoded via AJAX, WebSockets, etc. to peer 1
  7. Call startIce method of pc2 and send the generated candidate and more parameter to peer 1

    pc2.startIce()//will cause call iceCallback2 with candidate and more parametersfunction iceCallback2(candidate,more){
    //send candidate.toSdp() after URL encoding and more to peer 1
    }
  8. Accept answer at peer 1 candidate and more sent from peer 2 (steps 6 and 7), call startIce() and processIceMessage() methods of peerConnection. Send generated IceCandidate and more parameter to peer 2

    pc1.setRemoteDescription(pc1.SDP_ANSWER,decodeURIComponent(answer_sdp_encoded))
    pc1.startIce()
    //will cause call iceCallback1 with candidate and more parameters

    function iceCallback1(candidate,more){
    //send candidate.toSdp() after URL encoding and more to peer 1
    }
  9. call processIceMessage() method of the peerconnection objects (pc1, pc2) of both the peers

    pc1.processIceMessage(new IceCandidate(more,candidate));
    pc2.processIceMessage(new IceCandidate(more,candidate));

    //This will cause gotRemoteStream2 and gotRemoteStream1 to get executed function gotRemoteStream1(e){ //e.stream contains the audio and/or video stream of peer 2 vid.src=webkitURL.createObjectURL(e.stream) //obtain stream's URL and assign it to a video element } function gotRemoteStream2(e){ //e.stream contains the audio and/or video stream of peer 1 vid.src=webkitURL.createObjectURL(e.stream) //obtain stream's URL and assign it to a video element }
  10. NOP
I have created a webpage which can be used to test all the steps written above. It can be found on github .

Resources:





Thursday 30 August 2012

c# application that downloads facebook profiles pictures of members of a group

Getting started

There are quite a few steps involved in getting to display profile pictures of members of a  Facebook group.
There are two main parts to it:
  1. Getting data from Facebook
  2. Parsing it and displaying the result
Parsing the data returned from Facebook wasn't a trivial task since the data was JSON formatted and i had to try quite a few libraries to extract required fields from the returned JSON, C# having  no native support for JSON (though for silverlight a native library exists but that was of no use as this was a windows forms application ).
The library I used for parsing was LitJson. json.org  maintains a nice list of all the json parsing libraries available for various languages.

The following image shows the final application allong with the images fetched from one of my favourite facebook groups HITB .

Final Application's UI


Getting data from Facebook

For fetching data from facebook I used the facebooks Graph-API . I found the API simpler than its REST based API as as no complex http requests need to be made, you just fetch the required URL.
For getting the list of members in a group using the Graph Api the following url is used (more about the Group object here):

https://graph.facebook.com/groupid/members?access_token=accesstoken

where groupid is the id of the group and accestoken can be obtained from https://developers.facebook.com/tools/explorer 

once we have the data, we need to parse it to extract all the profile ids and user names. Once we have the profile Ids we can get the images easily as the url for a user's profile pic is nothing but

http://graph.facebook.com/profileID/picture

However there is a limitation to this method as the image returned by facebook is just 50x50 pixels so it might not be suitable in all scenarios.

For fetching the JSON fromated data containing info about the members of a group WebClient class can be used. Using the DownloadString() in  WebClient  object the entire JSON response can be stored in a string, which can later be fed to a JSON parser. Getting the images using webclient is not necessary since PictureBox controll can accept http paths as imagee source.

Parsing The data

The data returned by Facebook is in JSON but there is no native support for parsing JSON in C# thus we need to include an external library. After trying some libraries listed at json.org I found LitJson to be suitable for the project.

To add LitJson to the project download the package from http://litjson.sourceforge.net and extract it. Among the extracted directories the bin directory is the one which houses LitJson.dll. copy this dll to the project folder and add reference to it (RightClickProject->Addreference->Browse->selectDll->Ok) [this article explains the difference between Addreference and DllImport].
After adding reference include the LitJson namespace to the project by using the 'using' directive.
using LitJson;

To parse the JSON result we must first convert the stringified JSON obtained from WebClient to an object representing JSON. LitJson uses JsonData as a container class for JSON and JsonMapper.ToObject() to create the class from a string containing json.

Once the json object is created the 0th index of it will contain the data node (which contains the details of all the members) and 1st index contains paging which just gives the URL of the next page of the result(for a group having more than a certain number of users Facebook returns only a subset of the complete list in one page and we are required to iterate through pages to view all results, the logic for this is not implemented in the code and thus the application will not scan beyond the first page). so if jd is the object of type JsonData containing json data then jd[0] will be the data node of the returned json. now once we have the data node , obtaining is quite easy as the data for first user will be at jd[0][0] .
i.e.
jd[0]             :  data node
jd[0][i]        :  ith User's data
jd[0][i][0]  : ith User's name
jd[0][i][1]  : ith User's id
jd[0][i][2]  : whether the ith User is an administrator of the group


The Anchor attribute

While developing this application i stumbled upon this wonderful attribute that helps create a windows form application that allows the elements of the form to be automatically resized and/or reposition depending on the application size.

Anchor attribute

This MSDN article explains a different, albeit a better way to create resizable windows form.



Source Code

The complete source code along with binaries is available at Github (https://github.com/lihas/getProfilePicturesOfMembersOfaFacebookGroup).




related:
An article explaining how to attach a console to a windows form / gui application.

Monday 27 August 2012

Attaching console to Windows Forms Applicaion

This is a blog post that i have created for the only reason to remind myself at a later time that something like this is possible. At the end are various resources that help understand this concept better.

A windows form application and a console application are based on different subsystems. A subsystem is nothing but an environment in which an application runs. what that means basically is that when a console program has to put something on the monitor it will use a particular set of system calls and APIs while for a GUI program an entirely different collection of APIs may be required to draw something on screen.

The code used to attach a console to the windows form application is based around 'AllocConsole()' (see MSDN descrition) function. This function is defined in kernel32.dll and can be called by importing the dll file using DllImport attribute. The code for importing dll would be:


[DllImport("kernel32.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        static extern bool AllocConsole();

The extern attribute in the above given code is used to signal the presence of AllocConsole method in an external assembly i.e. some executable other that the current one. The Marshal attribute is used to bridge the data types used in .net framework (which are managed ) and external executables which uses unmanaged types. In this case since the kernel32 is implemented in c/c++  and the windows form application written in C# thus the return type BOOL of c/c++ must be bridged with bool of C#.
Marshalling and setting SetLastError fields in above code is not mandatory for this particular example and can be done away with however in scenarios where the  datatypes widely differ marshalling must be done.

Simply put the above code introduces the function AllocConsole present in kernel32.dll to the current scope (much like including header files) so that it can be called by rest of the program.
The above code must be written inside an enclosing class instead of a function and the method must be declared as static extern.

Now all we have to do is to  call 'AllocFunction()' at the required place. I included  it in the Form_Load method so that it remains available for the complete duration of the program. Although a feature like this is seldom required for a GUI application but during development and testing stage this proves to be a great alternative for debugging over message boxes and event logs.

It is important to note that the dll cannot be included by add-reference method as this dll is written in native language while the dlls that can be imported by add-reference method are the dlls written in .net thus DllImport is used. Also DllImport is not a method (unlike LoadLibrary or LoadLibraryEx in C++), instead it is an Attribute.

Some resources that are good for getting to know the windows subsystem better are:

  • These articles [(1) & (2)] on technet briefly explains the concept of GUI and CUI subsystems. 
  • MSDN blog explaining how OS knows which subsystem to use for a program.
  • about conhost.exe , articles 1 & 2.

Sunday 19 August 2012

Creating PowerPoint 2010 AddIn in c#

Creating PowerPoint Add-ins quite simple because of the nice documentation at MSDN and an extensive set of objects and methods which lets control each and every aspect of PowerPoint Interface.

The Add-in I created allows one to control a PowerPoint show with an android device.
The project thus consists of creating both an android app and the Add-in.
Since the two are created to work on different platforms thus an interface must be decided  that allows the two software to talk to each other. The interface I decided upon was a UDP connection over wifi in which the android app would send the 'number'/'id' of the button pressed and the Add-in would continuously check the sent number with a switch statement inside of an infinite while loop.

IMP: One thing to be careful of is that the addIn must not contain any long running code (such as those in socket programming) or sleep statements as the PowerPoint slideshow becomes non-responsive (addIn(s) are called in a blocking manner). So if such a code is necessary then a new thread must be created for it.

Creating Add-in consists of two main parts:

  1. Creating event handlers i.e. telling PowerPoint that this is the code you have to execute when certain things happen for eg. if you want to display a message box when 'slide-show' begins then you first create an event handler (viz just a function) and add the handler to the list of handlers associated with that event. so for our message box  example the code would be:
    • Application.SlideShowBegin += OnSlideShowBegin; //add OnSlideShowBegin to the list of functions that fire when slideshow begins.
    • void OnSlideShowBegin(SlideShowWindow Wn){MessageBox.Show('Slide Show Started');} //defining the code to be executed on function call
  2. Accessing the object model exposed by PowerPoint. What this means is that to interact with PowerPoint we need to access the various functions that are provided by the PowerPoint environment. For this there are various objects which are well documented on MSDN (here).
We start off by creating a new project in VC2010 and selecting PowerPoint 2010 Add-in template .
VS will automatically create the basic code structure containing a few event handlers. The Function
'private void ThisAddIn_Startup(object sender, System.EventArgs e)' is called on PowerPoint start-up. Since our Add-in is required to work only after the slide-show is started thus in the start-up function we add a function to the list of the functions executed when the slide-show starts. The 
code for it is:
      Application.SlideShowBegin += OnSlideShowBegin; //add a user defined                              function 'OnSlideShowBegin' to the list

In the definition of 'OnSlideShowBegin' we just create a new thread which will handle the socket connection. The 'OnSlideShowBegin' as an argument receives a 'SlideShowWindow' which contains the required methods to interact with the show. The functions required to change the slides, simulate mouse clicks , etc . can also be accessed by using the object. One good thing about the object is that it also gives access to the window's handle thus any other operation not directly supported by the API can be performed.


The android app is pretty self explanatory as there are only two parts to it:
  • connecting to Add-in's socket, and
  • sending the 'button number' on press of a button


The project was created by me in a duration of few hours and thus has quite a few Bugs. some of them are:
  • The App cannot connect to the Add-in if the slide show is started a second time, PowerPoint must be restarted. -BUG FIXED
  • The App and the Add-in have crashed quite a few times.
The safest way to make the Add-in work is by starting a new PowerPoint Application, opening the presentation and starting the slide show. Once the slide show has started the ip address and port of the computer running the presentation must be entered in the android app (separated by colon, for eg. 192.168.4.4:11000). As of now the port no 11000 is hard coded in the application and if it is to be changed , the application must be compiled again.

The code for the android app and the Add-in can be obtained from github at: 
the addin folder contains the c# code for the Addin and the Android App folder contains the code and other resources of the android app.

Additional Resources:
http://msdn.microsoft.com/en-us/library/office/ff743835.aspx

Wednesday 15 August 2012

Extract Imagex from WAIK (Windows Automted Installation Kit) without downloading the complete ISO

imagex is a component of Windows Automated Installation Kit (WAIK) which is required to customize windows installation images as in this post which explains creation of Windows 7 installation disk containing all versions of x86 and x64 architectures.

If you compare the size of imagex(about 500KB) to the size of WAIK(about 1.7GB!) It feels quit stupid to download the entire ISO for just that file. Also the download is not enough to get imagex.exe since WAIK contains setup file which must be installed to get imagex. So the overheads are Huge and hence in this blog I present a method to download just the minimum set of required files to run imagex.

software required:
  • HTTPDisk (can be downloaded from http://www.acc.umu.se/~bosse/ , just explore this site -contains lots of  "Awesome stuff"). What this software does is basically mounts a remote ISO i.e. an ISO file that is accessed over HTTP.
  • 7zip
Installation of HTTPDisk: It sure is an awesome software but it lacks enough documentation and community help i.e. if you run into some error then well, you are alone!

It took me quite a few hours to get it work. Initially I was trying to make it work on Win8 CP (x64)
but unfortunately I wasn't able to do that(I guess it is because I copied the 64bit driver in system32, I think I should have copied it in SySWOW64 instaed). however  in Windows 7 x32 it worked easily.
Do read the install.txt before following my steps.

For installing it on Win7 first I copied the httpdisk.sys file from "httpdisk-7 (1)\httpdisk-7\sys\obj\fre\i386" directory to system32\Drivers folder. Then I imported the httpdisk.reg file to registry and rebooted my system. After reboot I tried mounting the iso but it didn't work so I had to reboot again. After the second reboot the software worked just as expected.

To mount WAIK for win7 directly from Microsoft's Servers I used the following command:
  • httpdisk /mount 1 http://download.microsoft.com/download/8/E/9/8E9BBC64-E6F8-457C-9B8D-F6C9A16E6D6A/KB3AIK_EN.iso /cd l:
IMP.: In my case the number written after /mount switch is 1 as I had already mounted another iso and may be different depending on the number of iso files mounted. for none mounted before it would be 0.

    Now open the drive in which the iso was mounted, L in my case.

       The imagex file is present neutral.cab file in the iso (l:\neutral.cab).

    Open neutral.cab with 7z (right-click-open . do not download, it's a 500mb file !).
    The files F1_imagex, F2_imagex and F3_imagex  contain imagex.exe  for x32,IA64 and x64 architectures respectively , extract the required file to hard drive using 7zip's copy to dialogue and rename it to imagex.exe.

    This method can be used in all such scenarios where only a part of iso is required for example an iso containing  setup files for all architectures(x86,x64,IA64).

Making bootable disk containing all versions of Windows 7 of both x86 and x64 architecture (Works for Windows 10)


Recently I have learned about various Microsoft Offerings which ease the deployment of Operating systems for OEMs and also for any other environment involving large number of computers.
Some of them which are of particular interest  to me are:
  • WAIK (Windows automated Installation Kit- set of tools to customize and streamline OS install (1.7GB!)). [don't download it just yet, here is a way to save Bandwidth]
  • WDS (Windows Distribution Services- help deploy OS(s) over a network).
  • MDT (Microsoft Deployment Toolkit).

In a windows installation disk two files are of concern to us:
Now in an ordinary win7 installation disk if you remove the ei.cfg then all versions of Windows 7 are made available since ei.cfg is the file containing information about the version to install [more on ei.cfg at TechNet ].

The install.win file conatains the setup files for various versions OS. Ordinarily it will contain folders indexed from [1], [2], [3] …  and so on depending on the architecture (x86-32 has 5 versions while x86-64 has 4 versions coz of windows 7 starter being available on x86-32 architecture only). Along with the folders will be an XML file containing descriptions of all the images present in the install.wim file.

These files can be viewed once install.wim is opened with 7zip
 So to make available all versions of win7 of x86 and x64 architecture in the disk the corresponding folders must be added and entry be made in the XML file.

Before getting started we must have the following software:
  • win7 x86-32(32bit) installation medium
  • win7 x86-64(64 bit) installation medium
  • ei.cfg removal utility
  • imagex (part of WAIK (this post explains a more bandwidth friendly way to obtain imagex which doesn't require downloading and installing complete WAIK )
  • ultraiso or any other iso editing tool that allows creation of bootable disk.


Step1: Use ei.cfg removal utility to remove ei.cfg from iso (if you have installation disk just follow steps 2 and 3 and then remove ei.cfg from sources directory in 32bit and 64bit folders).

Step2: Make two directories in a drive containing enough free space (about 20gb) , lets call them  32bit and 64bit.

Step3: Extract the contents of x86-64 iso and x86-32 to 64bit and 32bit respectively.

Step4: Open elevated command prompt and navigate to the parent directory of 32bit and 64 bit folders . Also copy imagex to that directory. Now execute the following commands:
  • imagex /export 64bit\sources\install.wim 1 32bit\sources\install.wim "Windows 7 Home Basic x64"
  • imagex /export 64bit\sources\install.wim 2 32bit\sources\install.wim "Windows 7 Home Premium x64"
  • imagex /export 64bit\sources\install.wim 3 32bit\sources\install.wim "Windows 7 Professional x64"
  • imagex /export 64bit\sources\install.wim 4 32bit\sources\install.wim "Windows 7 Ultimate x64"
what the first command means is that from the source (64bit) install.wim move the first image (viz of win7 home basic) to install.wim of 32bit win7 (which already contains all the versions of win7-x32) and name it "Windows 7 Home Basic x64" or any other name which isn't already in the .wim file. This is not the name visible in the installation dialog. To change the name visible during installation, use imagex /info command.

  • imagex /info 32bit\sources\install.wim 4 "New_Image_Name" "New_Description"
New_Image_name should be unique, New_Description is the string which will appear during installation.
To delete an existing image use imagex /delete
  • imagex /delete 32bit\sources\install.wim image_index
The images in .wim file after image_index will shift 1 index up, eg. if image_index is 1, after this operation the image on index 2 will now come to index 1.



Step5 [Skip this step and go to Step 7 is creating bootable USB]: Using Ultra Iso extract boot image from the original Installation medium.
Step6 [Skip this step and go to Step 7 is creating bootable USB]: Create a new bootable disk in Ultra Iso select the extracted .bin file in step5 and paste all the contents of 32bit directory to it and save the ISO.

The Iso is ready now. During installation you the following dialogue will be visible, which can be then used to select the desired version to install.



Step 7 - Creating bootable USB: Ultra ISO is a paid software and trial version comes with limitations. If instead of ISO, you want to make a bootable USB drive do the following steps.

  1. Open elevated command prompt, and execute the following commands.

    diskpart
    list disk
    select disk 1 REM if list disk shows USB drive as disk 1
    clean
    create partition primary
    select partition 1
    active
    format fs=ntfs quick label="Win Install"
    exit
  2. In the same command prompt now execute the following commands

    cd 32bit\boot
    bootsect /nt60 d: REM assuming your USB drive is d:
    exit
  3. Copy all files from 32bit folder to USB device.

In the next blog I'll try to shorten the list of software required , particularly imagex ,ultraiso and ei.cfg removal utility.

Update (9 Nov 2018) : Tried the steps for Windows 10. Was successful .

References
  1. https://blogs.technet.microsoft.com/danstolts/2012/07/create-bootable-usb-thumb-drive-key-using-diskpart-with-windows-8-media-to-install-windows-8-release-preview-or-other-operating-system/