Category: Uncategorized

Epsilon greedy policy is a way of selecting random actions with uniform distribution from a set of available actions.
Using this policy either we can select random action with epsilon probability and we can select an action with 1-epsilon probability that gives maximum reward in given state. For example, if an experiment is about to run 10 times. This policy selects random actions in twice if the value of epsilon is 0.2.
Consider a following example, There is a robot with capability to move in 4 direction. Up,down,left,right. You can write a policy which selects a direction which gives maximum reward with 1-epsilon probability and with epsilon probability it select random direction. Here epsilon allows robot to explore new path.
Now the question is how to implement it.
Formula of estimated probability of an event is

probability of an event= number of times event occurs/( total number of trails) ;

We know the epsilon ( the probability of an event should occur) . And if we know the number of trails N then we can find number of times events should happen during N trails. so the pseudo code for implementing e-greedy policy can be written as

for(int i=0;i<nTrails;i++)

More sophisticated code can be written that selects random action for given probability with different permutation.But there is a one more elegant way to implement e-greedy policy.
Look at the code below and think does it make sense?

for (int i=0;i<nTrails;i++)
// function rand() returns random-number between 0 to 1 with uniform distribution
float temp=rand() ;

The trick is to use the classical probability definition. Definition says that

true probability of an event = (number of favorable outcomes of event)/ (number of possible outcomes) .
if number of trails approaches to infinity.

so the next question could be what is a probability that selected random-number is less than epsilon.
The answer is

total numbers which are less that epsilon/ ( total numbers ).

If a function rand() returns random number between 0 and 1 with interval of 0.1. for example 0,0.1,0.2,0.3,…0.8,0.9. and the value of epsilon is 0.2 then the estimated probability for selected number is less than 0.2 is (2/10)= 0.2 as per definition.
And probability of selecting random action is equal to the probability of selecting random number less than epsilon as per above code. So theoretically the code above selects the random action with epsilon probability.

We can verify the above argument by following code.
void greedy()

for (int i=0;i<nTrails,i++)
printf('probability of random action selected given no.trails %d is %f\n',nTrails,rn/nTrails);
Below is the output when function greedy calls for five times with nTrails equals to 1000 and 100000 . As it can be seen that number of trails increasing estimated probability is approaches to true probability.

probability of random action selected given no.trails 1000 is 0.214000
probability of random action selected given no.trails 1000 is 0.189000
probability of random action selected given no.trails 1000 is 0.164000
probability of random action selected given no.trails 1000 is 0.211000
probability of random action selected given no.trails 1000 is 0.216000

probability of random action selected given no.trails 100000 is 0.199320
probability of random action selected given no.trails 100000 is 0.198820
probability of random action selected given no.trails 100000 is 0.199450
probability of random action selected given no.trails 100000 is 0.198390
probability of random action selected given no.trails 100000 is 0.197960

Today I submitted Multi-touch driver of kinect for windows 7  to multi-touch vista project. Windows7 multitouch application now can be used with kinect.



OpenNI 1.3.2 or above and NITE or above ,Multi-TouchVista
How to install:
Extract files to folder “AddIns” . Select NITEProvider by running file Multitouch.Configuration.WPF.exe.
Wave Gesture to focus.
Grab Gesture To touch one need to do grab gesture and move one’s hand keeping fist closed.Open fist removes the contact from screen. Green dot means no contact to screen. Yellow means can now perform grab gesture to touch screen. Red means point is in contact with screen. Grab gesture should be done by facing hand to sensor.
Remark: Needs improvement in Grab Gesture. You need to first steady your hand then you can perform Grab Gesture.
Future work :Push Gesture need to be added for generating touch event instead of Grab Gesture.(Push Gesture is not support for multi handpoint by NITE in c#) .Code will be submitted soon.

FreeSWITCH tutorial part-2

In this tutorial I will show you how to configure  ivr_demo and Voice mail in German Language.You can configure even your dial plan.

1) Basic Setup

Install German sounds from freeswitch-sounds-de-at-ute-8000-0.0.1.tar.gz  from

extract in /opt/freeswitch/sounds.

In directory  “opt/freeswitch/sounds/de/at/ute/” create folder ivr and copy “8000” folder from “opt/freeswitch/sounds/de/at/ute/ivr_not_ok” . This step is important to enable ivr in German.

Install  freeswitch-lang-de  from synaptic if not installed.

Add ” <load module=”mod_say_de” />  ” in  /conf/autoconf/module.conf.xml  under ” <!– Say –> ” section.

In freeswitch.xml  add  ”  <X-PRE-PROCESS cmd=”include” data=”lang/de/*.xml”/> ”

<section name="phrases" description="Speech Phrase Management">
<X-PRE-PROCESS cmd="include" data="lang/de/*.xml"/>
<X-PRE-PROCESS cmd="include" data="lang/en/*.xml"/>

modify sound-path in /conf/lang/de/de.xml  to “$${sounds_dir}/de/at/ute”  .

<language name="de" sound-path="$${sounds_dir}/de/at/ute" tts-engine="cepstral" tts-voice="david">
<X-PRE-PROCESS cmd="include" data="demo/*.xml"/>
<!--voicemail_de_tts is purely implemented with tts, we need a files based implementation too -->

<X-PRE-PROCESS cmd=”include” data=”vm/tts.xml”/>


2) ivr_demo in German.

goto  /conf/dialplan/default.xml  .search for “ivr_demo” . when you call on 5000 ,you will hear  ivr in english language. To configure it in German add following line under “condition” section in “ivr_demo” extension.

<action application=”set” data=”default_language=de”/>

For example.

<!-- a sample IVR  -->
<extension name="ivr_demo">
<condition field="destination_number" expression="^5000$">
<action application="set" data="default_language=de"/>
<action application="answer"/>
<action application="sleep" data="2000"/>
<action application="ivr" data="demo_ivr"/>

copy /opt/freeswitch/conf/lang/en/demo/demo-ivr.xml to /opt/freeswitch/conf/lang/de/demo/ .

and modify /opt/freeswitch/conf/lang/de/de.xml to

<language name="de" sound-path="$${sounds_dir}/de/at/ute" tts-engine="cepstral" tts-voice="david">
    <X-PRE-PROCESS cmd="include" data="demo/*.xml"/>
<!--voicemail_de_tts is purely implemented with tts, we need a files based implementation too -->
<X-PRE-PROCESS cmd="include" data="vm/tts.xml"/>

IF you don’t follow above step you will get following error when you call on 5000

[ERR] switch_ivr_play_say.c:150 Can't find macro demo_ivr_main_menu.

[ERR] switch_ivr_play_say.c:150 Can’t find macro demo_ivr_main_menu_short


We are Done !!

3) Voicemail in German.

Open /conf/dialplan/default.xml  . Search <extension name=”vmain”> .

Add <action application=”set” data=”default_language=de”/>

For Example.

<!-- voicemail main extension -->
<extension name="vmain">
<condition field="destination_number" expression="^vmain$|^4000$|^\*98$">
<action application="set" data="default_language=de"/>
<action application="answer"/>
<action application="sleep" data="1000"/>
<action application="voicemail" data="check default ${domain_name}"/>

Call 4000 .

if you run into following error

" [ERR] switch_core_speech.c:61 Invalid speech module [cepstral]!
[ERR] switch_ivr_play_say.c:2136 Invalid TTS module!
[ERR] switch_core_speech.c:61 Invalid speech module [cepstral]!
[ERR] switch_ivr_play_say.c:2136 Invalid TTS module!
[ERR] switch_core_speech.c:61 Invalid speech module [cepstral]!
[ERR] switch_ivr_play_say.c:2136 Invalid TTS module!

You need to install cepstral Or you can follow the following trick.

copy /opt/freeswitch/conf/lang/en/vm/sounds.xml to /opt/freeswitch/conf/lang/de/vm/ .

modify /opt/freeswitch/conf/lang/de/de.xml to as follow.

<language name="de" sound-path="$${sounds_dir}/de/at/ute" tts-engine="cepstral" tts-voice="david">
<X-PRE-PROCESS cmd="include" data="demo/demo.xml"/>
<!--voicemail_de_tts is purely implemented with tts, we need a files based implementation too -->
   <X-PRE-PROCESS cmd="include" data="vm/tts.xml"/>
  <X-PRE-PROCESS cmd="include" data="vm/sounds.xml"/>

Call 4000 .

We are done!

3) custom dialplan in German.

In pervious tutorial we created simple dialplan which pronouns  some digits. dialplan was as follow.

<extension name="mydemo">
<condition field="destination_number" expression="^5343$">
<action application="answer"/>
<action application="say" data="en number pronounced 12345"/>

to pronouns digit in German you need to just add following line.

<extension name="mydemo">
<condition field="destination_number" expression="^5343$">
<action application="set" data="default_language=de"/>
<action application="answer"/>
<action application="say" data="en number pronounced 12345"/>

We are done!!

In this tutorial I am going to cover following topics.

1) Setting up freeswitch on Ubuntu.

2)Try demo-ivr application,voicemail

3) Create  simple dialplan.

1)Setting up freeswitch on Ubuntu.

        Install FreeSWITCH

Setting freeswitch on ubuntu is pretty  easy. Open terminal and execute following commands.
apt-get install python-software-properties
add-apt-repository ppa:freeswitch-drivers/freeswitch-nightly-drivers
apt-get update
apt-get install freeswitch freeswitch-lang-en freeswitch-sounds-en-us-callie-8000

You can also download  freeswitch-lang-de, freeswitch-lang-fr  etc to support different language.freeswitch will be installed in /opt directory.

Enabling FreeSWITCH

nano /etc/default/freeswitch


Start freeSWITCH.

execute invoke-rc.d freeswitch start or  sudo /opt/freeswitch/bin/freeswitch .

If you get  following error it means freeswitch is running in can try sudo /opt/freeswitch/bin/fs_cli.
Error: stacksize 4194303 is too large: run ulimit -s 240 or run /opt/freeswitch/bin/freeswitch -waste.
auto-adjusting stack size for optimal performance...
Cannot lock pid file /opt/freeswitch/run/
To stop freeswitch  execute invoke-rc.d freeswitch stop.

2) Try some sample demo IVR,voicemail,etc..

Go to /conf/diaplan/default.xml. Search for extension name  “ivr_demo”.  To try ivr_demo you need to call  on 5000 .

I assume you have started freeswitch in terminal and you have command prompt something like “freeswitch@user>”

Try following commands if you have not configured sip account.

load mod_portaudio

pa call 5000

pa hangup.

In /conf/dialplan/default.xml  search for  <extension name=”vmain”> . To try voicemail you need to call on 4000 .

3)Create simple dialplan

Go to /conf/diaplan/default.xml.  add following code

<extension name=”mydemo”>
<condition field=”destination_number” expression=”^5343$”>
<action application=”answer”/>
<action application=”say” data=”en number pronounced 12345″/>

in console execute   “reloadxml” and call on 5343 and see what happens.  please refer following link

Next Tutorial I will show how to configure freeswitch in German Language.

Here I am going to tell you how can you execute shell  script with super user permission from  android application.

Hmm… why do you need this at all ???

Hack the phone

Access the low level hardware

hm… hm… do whatever your instinct says !!

Understanding SU

Setup android sdk. now open shell  by

adb shell



SU is a process which runs with special permissions. when you type su , the  process starts and then you need to  type  shell command which you want  to execute from the console  like  “reboot” ,”del”, “echo 1 > sys/platform/lcd/enabled” ,”mount”.

Note that ” su reboot ” does not work because su does not take command line argument as process name to execute.

So basically what su does is it reads the command from command prompt and then creates child process ,executes it .

Basic Idea.

So the basic idea is when we  execute su  process  from the android application ,we  need to pass shell command or shell script to su process and then we need to stop su  by sending “exit” as we do with command prompt.

  Process p=Runtime.getRuntime().exec(“su”);

This  line starts su process. We can’t do “su reboot” . You know why :).  Now we have to send command name to  su process. when we do it  from shell  ,su reads  from some input stream.

  DataOutputStream  stream=new DataOutputStream(p.getOutputStream());

we have stream object which is connected to the input stream of process su . check out the following code.


stream.writeBytes(“exit \n”);


Reference: Gscript

Download source code and play with it. This project contains additional feature likes  how to read data from native process and use it in your application.


Kinect Cursor Control

When you get the kinect,the  first thing you want to do is to control your pc with with kinect.That’s what i did. well,its not that difficult.With c# ,OpenCv and Nite it is even more easy.Just Get the hand point and set the cursor. its done…!!!! but here is the problem,experience is not as good as mouse. Cursor is jerking and even more you have to move your hand more to reach the corner of the windows.

Check out the video.Here in video you see that with fine movement of hand  you can control the cursor easily. As you see cursor moment is pretty smooth and you don’t need to move your hands to the edge to reach the corner of the window.

Lets see how mouse works. well everyone know it sees the difference in movement and according to that it sets mouse position.we can do the same thing.But lets take different approach. we have to achieve two things.

  1. Control the cursor with your hand by moving hand in comfortable position to reach each corner of the window.
  2. Mouse movement should be smooth and controllable.

Addressing to the problem one , we can  see that we have to map  movement of hand in small space to bigger space.Basically,one can do linear mapping.We use the following notation in rest of the article.

rx=projective x of handpoint.

ry=projective y of handpoint.

cx=cursor x position.

cy=cursor y position.

rwidth=width of bounding box around hand point.

rheight=height of bounding box around hand point.

swidth=screen width.

sheight=screen height.

Let me elaborate “rwidth” and “rheight”. well what we do is we  define our comfort zone. lets say you  feel comfortable to move your hand 10 cm right and 10 cm left same as up and down to navigate within entire screen.Trick is when hand point is create we creat a virtual bounding box  of  ‘rwidth’ and ‘rheight’ keeping handpoint as a center. Now we are ready to go test. Try this.

cx=rx * (swidth/rwidth)


.It works but not so great.You don’t feel good. Cursor seems like jumping. Relation between cx and rx and between cy and ry is liner.


Solution to above problem is to make relationship between cx and rx and between cy and ry polynomial.

cx=pow(rx,n) *(swidth/pow(rwidth,n))


You can try now try with different n.  n=1.5 to n=3 works great. Following figures shows mapping for parameter rwidth=20 and  swidth=100 with different value of n.




Try with diffrent value of n see what happens.I dont think I need to explain graph you are smart enough to figure it out.

So finally mouse movemnt is somewhat good but….not satisfied. Cursor is still little bit jumpy.

Now lets address to second goal. Cursor movement should be smooth. We will use moving average . Get your self matlab and see how moving average works to smooth the curve. Here is the pseudo code for c#. Try to play with different windows size.  size=3 and n=1.7 gives better control.

int size=5;

int window_counter=0;

Point point_window[size];

void Point_update(Point HandPoint){ //method called by when hand point is updated





for(i=1; i<size-1;i++){









point_window[window_counter]=new Point(avgx,avgy);


Make computer to see the things the way you see.Reduce the dimensions.
To extract finger tips from hand, various techniques can be used like convex hull points, template matching etc..  One of the most difficult task it to extract hand from background. Kinect make it easy for us. The approach I used is somewhat  different to extract finger points.Check out the video. Green points show finger tips and red points show valley points.

Extracting hand is very tedious task but with kinnect hand point and depth image it is easy to do.lets see how  to do it.

Get kinect.Set up OpenNI and Nite. Download c# version of OpenCV. Play with PointViewer sample.Now you know  how to get HandPoint.. here is the step you can follow.

Background subtraction:

  • Get the hand point.
  • Get DepthMap.
  • Take a frame of   M x M pixels(say 200×200 px) from the DepthMap around hand point.
  • To subtract background  ,get depth of HandPoint from DepthMap. Make all pixel to value 0 whose depth is more or less than 5 cm and make rest of pixels to 1 in frame.


Here OpenCv save  a lot time. Extract contours using cvFindContour function and select the one contour which contains HandPoint. This contour is  our hand.

FingerTips Extraction:

So we have HandPoint and Hand contour. You can use convex hull  and convexity defect to extract desired  points and then try to identify finger tips.But it has many problems ,like you get more than  convex hull point near corner. So you need to add  filter to reduce it to one point for further processing.  and even more we are reading depthmap that is generated by IR projection.So in realtime video feed data we are getting changes over time ,in few frame point may be present and in few point may not be there.So our algorithme should able to cope this variation in image. Expirement your self you will know what I am talking about.

Polor cordinates:

We have image of lets say 200×200 px. what we will do is  make HandPoint as center of Cartesian system. Then we will find r and theta of each pixel.



Now   The idea is  values of r for each point is one dimention curve. find the local maxima and mimima of the local maxima is probable finger tips and mimima is propable valley points.

>Here we go.Get Sony PS 2 remote for your self. Don’t get afraid,just do it.Take a wire stripper ,cut a wire near by socket and strip it .Before we move foreword and connect this remote control to our ATmega16 or 8 whatever you have,lets have a closer look what each wire is for.

  • Brown :Data
  • Orange:Command
  • Gray:Not used
  • Black:Ground
  • Red:VCC
  • Yellow:ATT
  • Blue:Clock
  • White:Not used
  • Green:Ack

Wondered!! from where did I get this information!!! Well not a big deal.If you really want to find out something Google is always there for you :).
I found and
Now we connect ps2 remote control with Atmega16 as shown in following image.

Let me give you brief about how to fetch and send single byte data to PS2 remote control .

This image is taken from
Look at the image.It says that First we low the ATT pin to start communication with PS2 remote control.Then we start sending clock to receive and send data to PS2 remote control.we send different set of command using command line and receive data about switch and analog joystick from data line.we are sending and receving data byte wise. At the end of the communication we set ATT pin high. Data is put on the line by ps2 remote on the falling edge of the clock. we read the data on rising edge. If you want more detail check out above link .I don’t want write all that stuff again.
Questions!!! what are these commands ?? what kind of data ?? well answer is look at the protocol.

Analogue Controller in Red Mode

01     0x01   idle

02     0x42   0x79

03     idle    0x5A    Bit0 Bit1 Bit2 Bit3 Bit4 Bit5 Bit6 Bit7

04     idle    data    SLCT JOYR JOYL STRT UP   RGHT DOWN LEFT

05     idle    data    L2   R2   L1   R1   /\   O    X    |_|

06     idle    data    Right Joy 0x00 = Left  0xFF = Right

07     idle    data    Right Joy 0x00 = Up    0xFF = Down

08     idle    data    Left Joy  0x00 = Left  0xFF = Right

09     idle    data    Left Joy  0x00 = Up    0xFF = Down

we first low the ATT and then we send three synchronization command byte1,byte2,byte3 and then from the 4th byte we start receiving data from ps2 remote control as described in protocol. store these data for latter use.after the 9th byte recevied we again make ATT to high .we process over the stored data and again start looking for new data as described above.

Lets do some coding .I hope You are familiar micro controller coding and have play around it.Here is the code for Atmega16,8.
send_cmd(0x1,1); //start
send_cmd(0x42,2); //id
send_cmd(0x00,3); //incoming indication
send_cmd(0x00,4); //get 4th byte
send_cmd(0x00,5); //5th byte
send_cmd(0x00,6); //6th byte
send_cmd(0x00,7); //7th byte
send_cmd(0x00,8); //8th byte
send_cmd(0x00,9); //9 th byte
PORTB |=_BV(alt); //clear alt

Simple aha!!. Before we look at send_cmd function lets see following problem.
int val=0x23;
how do you put 0x23 value serially on any pin of microcontroller.
binary value of 0x23 is 0010 0011.

int mask=1;
for(int i=0;i<8;i++){
mask=1 << i ;
if (mask & val )


Make Sense!!!

Now its time to jump in to send_cmd function.

void send_cmd(unsigned char ps2cmd,int num)
unsigned char mask;
int k;

PORTB |= _BV(cmd);

PORTB &=~_BV(cmd);
mask <<=1;
PORTB &=~_BV(clk);

PORTB |=_BV(clk);



if(ps2cmd==0x0) {
case 4:


case 5:
case 6:
case 7:
case 8:

case 9:



Do You Really Need explanation of the code..No you are smart enough to figure it out. 🙂

>Killing Summer Boredom

>Summer 2008 had been great fun. This year I joined the ROBOCON Team. We(ROBOCON Team of L.D. College of Engineering) went to Pune to participate in the competition. We just came back after the great trip to Pune.While arranging the things back at the ROBOCON office, I found Sony PS2 remote. I used to drive manual robot with Sony PS2 remote in ROBOCON. The idea of using this remote to play the games excited me. The only step left for me was to attach it with my laptop using USB. Believe me, USB is not that simple as it seems to be. It came up as one of the most complicated protocol I have ever come across. Those days I was playing around ATMEL ATmega8,16,32 and that time these were the best controller we could get.After searching for few days , I found V-USB. to implement USB protocol. Don’t get scared but ya this was my first recording…!!

NFS Underground2 gave me enough motivation to make a small device so that it looks compact. After settling all the hardware components in small device, I was ready to give my second shot…!!! Now here It is , final presentation with Prince of Persia….!!!!
Please check other post to understand how it works and to download source.