When you get the kinect,the  first thing you want to do is to control your pc with with kinect.That’s what i did. well,its not that difficult.With c# ,OpenCv and Nite it is even more easy.Just Get the hand point and set the cursor. its done…!!!! but here is the problem,experience is not as good as mouse. Cursor is jerking and even more you have to move your hand more to reach the corner of the windows.

Check out the video.Here in video you see that with fine movement of hand  you can control the cursor easily. As you see cursor moment is pretty smooth and you don’t need to move your hands to the edge to reach the corner of the window.

Lets see how mouse works. well everyone know it sees the difference in movement and according to that it sets mouse position.we can do the same thing.But lets take different approach. we have to achieve two things.

  1. Control the cursor with your hand by moving hand in comfortable position to reach each corner of the window.
  2. Mouse movement should be smooth and controllable.

Addressing to the problem one , we can  see that we have to map  movement of hand in small space to bigger space.Basically,one can do linear mapping.We use the following notation in rest of the article.

rx=projective x of handpoint.

ry=projective y of handpoint.

cx=cursor x position.

cy=cursor y position.

rwidth=width of bounding box around hand point.

rheight=height of bounding box around hand point.

swidth=screen width.

sheight=screen height.

Let me elaborate “rwidth” and “rheight”. well what we do is we  define our comfort zone. lets say you  feel comfortable to move your hand 10 cm right and 10 cm left same as up and down to navigate within entire screen.Trick is when hand point is create we creat a virtual bounding box  of  ‘rwidth’ and ‘rheight’ keeping handpoint as a center. Now we are ready to go test. Try this.

cx=rx * (swidth/rwidth)

cy=ry*(sheight/rheight)

.It works but not so great.You don’t feel good. Cursor seems like jumping. Relation between cx and rx and between cy and ry is liner.

x1=x2

Solution to above problem is to make relationship between cx and rx and between cy and ry polynomial.

cx=pow(rx,n) *(swidth/pow(rwidth,n))

cy=pow(ry,n)*(sheight/pow(rheight,n)).

You can try now try with different n.  n=1.5 to n=3 works great. Following figures shows mapping for parameter rwidth=20 and  swidth=100 with different value of n.

n=1.7

n=2

n=3

Try with diffrent value of n see what happens.I dont think I need to explain graph you are smart enough to figure it out.

So finally mouse movemnt is somewhat good but….not satisfied. Cursor is still little bit jumpy.

Now lets address to second goal. Cursor movement should be smooth. We will use moving average . Get your self matlab and see how moving average works to smooth the curve. Here is the pseudo code for c#. Try to play with different windows size.  size=3 and n=1.7 gives better control.

int size=5;

int window_counter=0;

Point point_window[size];

void Point_update(Point HandPoint){ //method called by when hand point is updated

cx=pow(rx,n)*swidth/pow(rwidth,n);

cy=pow(ry,n)*rheight/pow(rheight,n);

avgx=cx;

avgy=cy;

for(i=1; i<size-1;i++){

avgx+=point_window[i].x;

avgy+=point_window[i].y;

}

avgx=avgx/size;

avgy=avgy/size;

if(window_counter>(size-1)){

window_counter=0;

}

point_window[window_counter]=new Point(avgx,avgy);

}