NeuroPy : Python library for interfacing with Neurosky's Mindwave eeg headset

Neurosky Mindwave is a miniature eeg machine which i used in a project to control a harware based on our thoughts. It involved taking eeg data from the headset and then performing certain actions based on it.
Neurosky provides a lot of developmental resources and sdks and libraries for various platforms.
The support for Python However was missing.

To obtain data from the headset using python i had to write my own library, NeuroPy  based on the excellent description of the protocol provided by the company (google for mindset_communications_protocol).
The headset establishes a spp (serial port profile) connection over Bluetooth thus receiving and transmitting data is as easy as communicating with any other serial port.

The packet structure is fairly easy to understand from the document thus any description of it is not given here. However using the library is explained.

imp. :this library need pyserial for it's operation, thus make sure that dependencies are satisfied. (it also requires thread module but that is available in a standard python installation, pyserial can be downloaded from PyPi)

Installation

1. Download the source distribution (zip file) from dist directory
2. unzip and navigate to the folder containing setup.py and other files
3. run the following command: python setup.py install

There are three main steps involved in connecting the headset to the mindwave headset then a subsequent 4th step which is explained below can be used to fetch data from the device.

1. Pairing the device
   
   In windows the bluetooth device can be paired by clicking 'add new device' in the 'devices and printers' settings dialogue. Mindwave will reserve a virtual COM port for it, such as COM1 or COM2. To know which port is reserved check properties of mindwave, which will be visible in 'devices and printers' section
   
   In linux tools such as rfcomm can be used to pair. After pairing take a note of the device in the /dev/ directory allocated to the headset. I paired the headset using information given on the following links:
             askubuntu.com and
             westernwillow.com
2. Once the device is paired the library can be initialised in the following manner:
   object1=NeuroPy("COM6",57600) #windows
3. The library provides data in two ways, either via callback mechanism or via accessing the required properties (variables for eg. object1.attention to get value for attention). This is explained in step 4.
   If required the callbacks are set and the start method is called.
   object1.start()
4. The data from the device can be obtained using either of the following methods or both of them together:
   
   a) Obtaining value: variable1=object1.attention #to get value of attention
   
   #other variables: attention,meditation,rawValue,delta,theta,lowAlpha,highAlpha,lowBeta,highBeta,lowGamma,midGamma, poorSignal and blinkStrength
   
   b) Setting callback:a call back can be associated with all the above variables so that a function is called when the variable is updated.
   Syntax: setCallBack("variable",callback_function)
   for eg. to set a callback for attention data the syntax will be setCallBack("attention",callback_function)

Sample Program

from NeuroPy import NeuroPy
object1=NeuroPy("COM6") #If port not given 57600 is automatically assumed
                        #object1=NeuroPy("/dev/rfcomm0") for linux
def attention_callback(attention_value):
    "this function will be called everytime NeuroPy has a new value for attention"
    print "Value of attention is",attention_value
    #do other stuff (fire a rocket), based on the obtained value of attention_value
    #do some more stuff
    return None

#set call back:
object1.setCallBack("attention",attention_callback)

#call start method
object1.start()

while True:
    if(object1.meditation>70): #another way of accessing data provided by headset (1st being call backs)
        object1.stop()         #if meditation level reaches above 70, stop fetching data from the headset

Links:

• Github repo: https://github.com/lihas/NeuroPy
• Source Distribution Package (installation package for python): https://github.com/lihas/NeuroPy/tree/master/dist
• Package can also be downloaded From PyPi: https://pypi.python.org/pypi/NeuroPy/0.1

Capturing packets from an Android device and analyzing them on pc in real time

Capturing network packets on Android and analysing them in real time is not a trivial task. Various off the shelf apps exist for capturing packets however none of them comes close to tools available for pc such as Wireshark. Most of them just dump the packets to a file and aren't of much use.

This post explores one of the ways to analyse packets from an android device on a pc. This article on mobile.tutsplus.com is the place from where much of the work has been drawn from. The reason for making a separate post is because i couldn't get the capture to work by exactly following the directions given on the aforementioned website and i had to make a few changes of my own.

The following are the steps by which packets are captured from an android device and then sent to a pc for analysis.

1. capture packets using tcpdump
2. pipe the packets to a netcat server (which basically means that we are sending the packets to a netcat server from where they can be fetched by a client connecting to the server).
3. using adb forward command forward the remote port (the port on which netcat server is listening) of the android device  to a local port of the PC (wiz making a port on the droid available as a port on pc) .
   the syntax of adb forward command is:
   adb forward <local_port (on PC) > <remote_port (on droid) >
4. connecting a netcat client to the forwarded port on PC and getting the packets.
5. piping / routing the packets from netcat to wireshark.

before getting started arm images of tcpdump and netcat must be present on the droid. (did i mention that the droid must be rooted?)

to check if you have correct images file command can be used (on Linux, windows users might use web services such as virus total as they also give the description of the file along with virus scan report).

For getting help with installing these tools on droid the mobile.tutsplus.com article can be referenced.

Once we are ready with all the tools the following steps haave to be followed:

1. obtain a shell on device (adb shell) and switch to root user (su).
2. execute the following command
   /data/local/tcpdump -n -s 0 -w - | /data/local/nc -l -p 12345 (on my droid tcpdump and netcat executables were present in /data/local)
3. forward the port 12345 (port on which netcat is listening) of the droid to port number 54321 on the PC. The command for doing this is:
     adb forward tcp:54321 tcp:12345
4. connect the netcat client on pc to local port 54321 using the following command and redirect its output to /dev/null (i am on a linux machine).
    nc 127.0.0.1 54321 > /dev/null
5. open wireshark and start capturing packets on the local loopback interface (not awailable on windows).
6. Apply the following filter in wireshark to limit displaying of packets from port 54321 only
   tcp.port eq 54321
7. voila! the packets can now be seen in wireshark.

My system configuration:
OS: ubuntu 12.04
android device: micromax canvas 2 (a110) running android 4.0.4 on a dual core processor ;)

Some of the problems i faced and their solutions:

when using a named pipe by creating one with mkfifo command , piping netcat's output to this pipe and then reading from this pipe in wire shark (using -i switch) the packets were not seen by wireshark as TCP packets however when i tried the above mentioned method , it worked correctly.

Next challenge :
When using facebook chat or whatsapp the packets weren't shown in wireshark, maybe tcpdump is at fault. so the next challenge would be to somehow analyse packets from these two apps.
Also since all the output is obtained from one single port, wireshark sees the all the packets as a part of one giant TCP connection.

WebRTC - using Media Stream and PeerConnection API

WebRTC is a great new addition to the HTML5 spec. It gives developers the ability to make apps that can process the audio and video feeds of a user with utmost ease. So instead of handling video drivers, bandwidth, encryption, etc. the developer needs to call simple javascript APIs. It also gives the assurance that the application is going to work on different platform (As usual while all browser vendors are struggling to integrate most of the specification into their browser, Microsoft has come up with it's own implementation).

* The code i have written works on chrome after enabling the flags which are required to make WebRTC work. Also since the WebRTC spec isn't finalised most of the code uses vendor prefixes. 

Creating a P2P audio-video application using WebRTC is a two step process. The first is to ask the user for permissions and get the audio/video feed. The second is streaming and/or processing the feed.

1. Getting the feed

The feed can be obtained by calling getUserMedia (webkitGetUserMedia) of the navigator object.

following is the code which will ask for permission to access audio and video feed (camera and mic) of a user.

navigator.webkitGetUserMedia({audio:true,video:true},onSuccessFunction,onErrorFunction)

The function accepts three parameters. The first is an object containing the list of devices that are required, second argument is the name of the function that will be called on success i.e. when user grants the request and there was no problem while allocating devices and the third argument is the name of the function that gets called when the request fails.

onSuccessFunction(strm){

strm//object containing requested streams

}

2. Processing the feed

Once the stream object is obtained it can be displayed locally in a <video> element, converted to binary data using canvas and then transmitted over websockets for server-side processing , streamed to another machine using peerConnection API.

To display the feed in a video element the stream-object has to be converted to a url.

var url=webkitURL.createObjectURL(strm)

The obtained url can now be assigned to a <Video> element's src attribute.

video1.src=url

video1.play() //stream starts playing

To convert video stream to binary data each frame of the video steam must be displayed in canvas by either using <video> element or using Image() object of javascript and then calling drawimage() on canvas's context.

The binary data can then be obtained by either using toDataUrl() or getImageData() methods of canvas or canvas's context respectively.

Using PeerConnection API :

Peer Connection API lies at the heart of real time communication . It allows P2P (browser-to-browser) streaming of audio and video information. The setup of peer-connection takes place by exchanging some data termed as offer, response and IceCandidates. The object containing the initial request generated by a peer is know as offer, the corresponding answer is know as 'answer'. After exchange of offer and answer startIce method of peer-connection is called and the generated IceCandidates are exchanged (not sure why IceCandidate is necessary) . The following graphic explains the setup for peerconnection.

IMP: While sending offer, answer and candidate the entire object need not (read cannot as i know of no method to send javascript objects) be sent. Instead offer and answer can be converted to SDP by using offer.toSdp() and answer.toSdp(), now since SDP is nothing but a string , it thus can be sent easily after encoding by encodeURIComponent() javascript method. Also it is better to use semicolons to terminate javascript statements .

Peer Connection process:

1. create new peer connection object.
   
   pc1=new webkitPeerConnection00("STUN stun.l.google.com:19302",iceCallback1)
   
   //first argument: stun server, 2nd argument: name of function called when startIce() is called
   //as of now the function name is webkitPeerConnection00, however PeerConnection would be the final name, 00 is for compatibility with the earlier and now depreciated version of peer connection
   
2. add callbacks and audio and/or video stream.
   pc1.onaddstream=gotRemoteStream1 //function to be called when remote stream is obtained
   pc1.addStream(localstream1) //add the stream object obtained from navigator.getUserMedia()
   
3. Repeat the same process for other peer.
4. Create offer object at peer 1 (peer 1=peer that initiates connection, peer 2=the other peer) and set local-description.
   
   var offer=pc1.createOffer(null) //not sure about the argument accepted by this method and why this is null
   pc1.setLocalDescription(pc.SDP_OFFER,offer)
   
5. Send SDP of offer object to peer 2, after URL encoding
   var offer_sdp_encoded=encodeURIComponent(offer.toSdp())
   //send offer_sdp_encoded via AJAX, WebSockets, etc. to peer 1
   
6. Accept offer at peer 2, set remote and local descriptions and send the 'answer' to peer 1
   
   pc2=new webkitPeerConnection00("STUN stun.l.google.com:19302",iceCallback2) pc2.onaddstream=gotRemoteStream2 pc2.setRemoteDescription(pc2.SDP_OFFER,new SessionDescription(decodeURIComponent(offer_sdp_encoded))) //offer_sdp_encoded recieved via AJAX, WebSockets, etc. from peer 1
   
   var answer=pc2.createAnswer(decodeURIComponent(offer_sdp_encoded),{has_audio:true,has_video:true})
   pc2.setLocalDescription(pc2.SDP_ANSWER,answer)
   
   answer_sdp_encoded=encodeURIComponent(answer.tosdp())
   //send answer_sdp_encoded via AJAX, WebSockets, etc. to peer 1
   
7. Call startIce method of pc2 and send the generated candidate and more parameter to peer 1
   
   pc2.startIce()//will cause call iceCallback2 with candidate and more parametersfunction iceCallback2(candidate,more){
   //send candidate.toSdp() after URL encoding and more to peer 1
   }
   
8. Accept answer at peer 1 candidate and more sent from peer 2 (steps 6 and 7), call startIce() and processIceMessage() methods of peerConnection. Send generated IceCandidate and more parameter to peer 2
   
   pc1.setRemoteDescription(pc1.SDP_ANSWER,decodeURIComponent(answer_sdp_encoded))
   pc1.startIce() //will cause call iceCallback1 with candidate and more parameters
   
   function iceCallback1(candidate,more){
   //send candidate.toSdp() after URL encoding and more to peer 1
   }
   
9. call processIceMessage() method of the peerconnection objects (pc1, pc2) of both the peers
   
   pc1.processIceMessage(new IceCandidate(more,candidate));
   pc2.processIceMessage(new IceCandidate(more,candidate));
   
   //This will cause gotRemoteStream2 and gotRemoteStream1 to get executed function gotRemoteStream1(e){ //e.stream contains the audio and/or video stream of peer 2 vid.src=webkitURL.createObjectURL(e.stream) //obtain stream's URL and assign it to a video element } function gotRemoteStream2(e){ //e.stream contains the audio and/or video stream of peer 1 vid.src=webkitURL.createObjectURL(e.stream) //obtain stream's URL and assign it to a video element }
10. NOP

I have created a webpage which can be used to test all the steps written above. It can be found on github .

Resources:

• Google I/O video
• html5rocks.com
• wikipedia: about STUN servers and ICE protocols.
• Peer Connection Test Webpage

c# application that downloads facebook profiles pictures of members of a group

Getting started

There are quite a few steps involved in getting to display profile pictures of members of a  Facebook group.

There are two main parts to it:

1. Getting data from Facebook
2. Parsing it and displaying the result

Parsing the data returned from Facebook wasn't a trivial task since the data was JSON formatted and i had to try quite a few libraries to extract required fields from the returned JSON, C# having  no native support for JSON (though for silverlight a native library exists but that was of no use as this was a windows forms application ).

The library I used for parsing was LitJson. json.org  maintains a nice list of all the json parsing libraries available for various languages.

The following image shows the final application allong with the images fetched from one of my favourite facebook groups HITB .

Getting data from Facebook

For fetching data from facebook I used the facebooks Graph-API . I found the API simpler than its REST based API as as no complex http requests need to be made, you just fetch the required URL.

For getting the list of members in a group using the Graph Api the following url is used (more about the Group object here):

https://graph.facebook.com/groupid/members?access_token=accesstoken

where groupid is the id of the group and accestoken can be obtained from https://developers.facebook.com/tools/explorer 

once we have the data, we need to parse it to extract all the profile ids and user names. Once we have the profile Ids we can get the images easily as the url for a user's profile pic is nothing but

http://graph.facebook.com/profileID/picture

However there is a limitation to this method as the image returned by facebook is just 50x50 pixels so it might not be suitable in all scenarios.

For fetching the JSON fromated data containing info about the members of a group WebClient class can be used. Using the DownloadString() in  WebClient  object the entire JSON response can be stored in a string, which can later be fed to a JSON parser. Getting the images using webclient is not necessary since PictureBox controll can accept http paths as imagee source.

Parsing The data

The data returned by Facebook is in JSON but there is no native support for parsing JSON in C# thus we need to include an external library. After trying some libraries listed at json.org I found LitJson to be suitable for the project.

To add LitJson to the project download the package from http://litjson.sourceforge.net and extract it. Among the extracted directories the bin directory is the one which houses LitJson.dll. copy this dll to the project folder and add reference to it (RightClickProject->Addreference->Browse->selectDll->Ok) [this article explains the difference between Addreference and DllImport].

After adding reference include the LitJson namespace to the project by using the 'using' directive.

using LitJson;

To parse the JSON result we must first convert the stringified JSON obtained from WebClient to an object representing JSON. LitJson uses JsonData as a container class for JSON and JsonMapper.ToObject() to create the class from a string containing json.

Once the json object is created the 0th index of it will contain the data node (which contains the details of all the members) and 1st index contains paging which just gives the URL of the next page of the result(for a group having more than a certain number of users Facebook returns only a subset of the complete list in one page and we are required to iterate through pages to view all results, the logic for this is not implemented in the code and thus the application will not scan beyond the first page). so if jd is the object of type JsonData containing json data then jd[0] will be the data node of the returned json. now once we have the data node , obtaining is quite easy as the data for first user will be at jd[0][0] .

i.e.

jd[0]             :  data node

jd[0][i]        :  i^th User's data
jd[0][i][0]  : i^th User's name
jd[0][i][1]  : i^th User's id
jd[0][i][2]  : whether the i^th User is an administrator of the group


The Anchor attribute

While developing this application i stumbled upon this wonderful attribute that helps create a windows form application that allows the elements of the form to be automatically resized and/or reposition depending on the application size.

Anchor attribute

This MSDN article explains a different, albeit a better way to create resizable windows form.

Source Code

The complete source code along with binaries is available at Github (https://github.com/lihas/getProfilePicturesOfMembersOfaFacebookGroup).




related:
An article explaining how to attach a console to a windows form / gui application.

Attaching console to Windows Forms Applicaion

This is a blog post that i have created for the only reason to remind myself at a later time that something like this is possible. At the end are various resources that help understand this concept better.

A windows form application and a console application are based on different subsystems. A subsystem is nothing but an environment in which an application runs. what that means basically is that when a console program has to put something on the monitor it will use a particular set of system calls and APIs while for a GUI program an entirely different collection of APIs may be required to draw something on screen.

The code used to attach a console to the windows form application is based around 'AllocConsole()' (see MSDN descrition) function. This function is defined in kernel32.dll and can be called by importing the dll file using DllImport attribute. The code for importing dll would be:


[DllImport("kernel32.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        static extern bool AllocConsole();

The extern attribute in the above given code is used to signal the presence of AllocConsole method in an external assembly i.e. some executable other that the current one. The Marshal attribute is used to bridge the data types used in .net framework (which are managed ) and external executables which uses unmanaged types. In this case since the kernel32 is implemented in c/c++  and the windows form application written in C# thus the return type BOOL of c/c++ must be bridged with bool of C#.
Marshalling and setting SetLastError fields in above code is not mandatory for this particular example and can be done away with however in scenarios where the  datatypes widely differ marshalling must be done.

Simply put the above code introduces the function AllocConsole present in kernel32.dll to the current scope (much like including header files) so that it can be called by rest of the program.
The above code must be written inside an enclosing class instead of a function and the method must be declared as static extern.

Now all we have to do is to  call 'AllocFunction()' at the required place. I included  it in the Form_Load method so that it remains available for the complete duration of the program. Although a feature like this is seldom required for a GUI application but during development and testing stage this proves to be a great alternative for debugging over message boxes and event logs.

It is important to note that the dll cannot be included by add-reference method as this dll is written in native language while the dlls that can be imported by add-reference method are the dlls written in .net thus DllImport is used. Also DllImport is not a method (unlike LoadLibrary or LoadLibraryEx in C++), instead it is an Attribute.

Some resources that are good for getting to know the windows subsystem better are:

• These articles [(1) & (2)] on technet briefly explains the concept of GUI and CUI subsystems. 
• MSDN blog explaining how OS knows which subsystem to use for a program.
• about conhost.exe , articles 1 & 2.

Creating PowerPoint 2010 AddIn in c#

Creating PowerPoint Add-ins quite simple because of the nice documentation at MSDN and an extensive set of objects and methods which lets control each and every aspect of PowerPoint Interface.

The Add-in I created allows one to control a PowerPoint show with an android device.
The project thus consists of creating both an android app and the Add-in.
Since the two are created to work on different platforms thus an interface must be decided  that allows the two software to talk to each other. The interface I decided upon was a UDP connection over wifi in which the android app would send the 'number'/'id' of the button pressed and the Add-in would continuously check the sent number with a switch statement inside of an infinite while loop.

IMP: One thing to be careful of is that the addIn must not contain any long running code (such as those in socket programming) or sleep statements as the PowerPoint slideshow becomes non-responsive (addIn(s) are called in a blocking manner). So if such a code is necessary then a new thread must be created for it.

Creating Add-in consists of two main parts:

1. Creating event handlers i.e. telling PowerPoint that this is the code you have to execute when certain things happen for eg. if you want to display a message box when 'slide-show' begins then you first create an event handler (viz just a function) and add the handler to the list of handlers associated with that event. so for our message box  example the code would be:
• Application.SlideShowBegin += OnSlideShowBegin; //add OnSlideShowBegin to the list of functions that fire when slideshow begins.
• void OnSlideShowBegin(SlideShowWindow Wn){MessageBox.Show('Slide Show Started');} //defining the code to be executed on function call
2. Accessing the object model exposed by PowerPoint. What this means is that to interact with PowerPoint we need to access the various functions that are provided by the PowerPoint environment. For this there are various objects which are well documented on MSDN (here).

We start off by creating a new project in VC2010 and selecting PowerPoint 2010 Add-in template .

VS will automatically create the basic code structure containing a few event handlers. The Function

'private void ThisAddIn_Startup(object sender, System.EventArgs e)' is called on PowerPoint start-up. Since our Add-in is required to work only after the slide-show is started thus in the start-up function we add a function to the list of the functions executed when the slide-show starts. The 

code for it is:

      Application.SlideShowBegin += OnSlideShowBegin; //add a user defined                              function 'OnSlideShowBegin' to the list

In the definition of 'OnSlideShowBegin' we just create a new thread which will handle the socket connection. The 'OnSlideShowBegin' as an argument receives a 'SlideShowWindow' which contains the required methods to interact with the show. The functions required to change the slides, simulate mouse clicks , etc . can also be accessed by using the object. One good thing about the object is that it also gives access to the window's handle thus any other operation not directly supported by the API can be performed.

The android app is pretty self explanatory as there are only two parts to it:

• connecting to Add-in's socket, and
• sending the 'button number' on press of a button

The project was created by me in a duration of few hours and thus has quite a few Bugs. some of them are:

• The App cannot connect to the Add-in if the slide show is started a second time, PowerPoint must be restarted. -BUG FIXED
• The App and the Add-in have crashed quite a few times.

The safest way to make the Add-in work is by starting a new PowerPoint Application, opening the presentation and starting the slide show. Once the slide show has started the ip address and port of the computer running the presentation must be entered in the android app (separated by colon, for eg. 192.168.4.4:11000). As of now the port no 11000 is hard coded in the application and if it is to be changed , the application must be compiled again.

The code for the android app and the Add-in can be obtained from github at: 

https://github.com/lihas/PowerPointAddin

the addin folder contains the c# code for the Addin and the Android App folder contains the code and other resources of the android app.

Additional Resources:
http://msdn.microsoft.com/en-us/library/office/ff743835.aspx

Extract Imagex from WAIK (Windows Automted Installation Kit) without downloading the complete ISO

imagex is a component of Windows Automated Installation Kit (WAIK) which is required to customize windows installation images as in this post which explains creation of Windows 7 installation disk containing all versions of x86 and x64 architectures.

If you compare the size of imagex(about 500KB) to the size of WAIK(about 1.7GB!) It feels quit stupid to download the entire ISO for just that file. Also the download is not enough to get imagex.exe since WAIK contains setup file which must be installed to get imagex. So the overheads are Huge and hence in this blog I present a method to download just the minimum set of required files to run imagex.

software required:

• HTTPDisk (can be downloaded from http://www.acc.umu.se/~bosse/ , just explore this site -contains lots of  "Awesome stuff"). What this software does is basically mounts a remote ISO i.e. an ISO file that is accessed over HTTP.
• 7zip

Installation of HTTPDisk: It sure is an awesome software but it lacks enough documentation and community help i.e. if you run into some error then well, you are alone!

It took me quite a few hours to get it work. Initially I was trying to make it work on Win8 CP (x64)
but unfortunately I wasn't able to do that(I guess it is because I copied the 64bit driver in system32, I think I should have copied it in SySWOW64 instaed). however  in Windows 7 x32 it worked easily.
Do read the install.txt before following my steps.

For installing it on Win7 first I copied the httpdisk.sys file from "httpdisk-7 (1)\httpdisk-7\sys\obj\fre\i386" directory to system32\Drivers folder. Then I imported the httpdisk.reg file to registry and rebooted my system. After reboot I tried mounting the iso but it didn't work so I had to reboot again. After the second reboot the software worked just as expected.

To mount WAIK for win7 directly from Microsoft's Servers I used the following command:

• httpdisk /mount 1 http://download.microsoft.com/download/8/E/9/8E9BBC64-E6F8-457C-9B8D-F6C9A16E6D6A/KB3AIK_EN.iso /cd l:

IMP.: In my case the number written after /mount switch is 1 as I had already mounted another iso and may be different depending on the number of iso files mounted. for none mounted before it would be 0.

Now open the drive in which the iso was mounted, L in my case.

       The imagex file is present neutral.cab file in the iso (l:\neutral.cab).

Open neutral.cab with 7z (right-click-open . do not download, it's a 500mb file !).
The files F1_imagex, F2_imagex and F3_imagex  contain imagex.exe  for x32,IA64 and x64 architectures respectively , extract the required file to hard drive using 7zip's copy to dialogue and rename it to imagex.exe.

This method can be used in all such scenarios where only a part of iso is required for example an iso containing  setup files for all architectures(x86,x64,IA64).