TAFFI

So, this TAFFI thing is pretty simple, actually. It stands for Thumb And Fore Finger Interface, a new gestural input method developed by Andrew D. Wilson, a Microsoft Researcher. So basically, the algorithm is able to detect the pinching gesture when you bring your thumb and fore finger together, and gives you its coordinates, headings, size, etc. The amazing thing about it is that it doesn't use any fancy hand recognition method, instead, it uses a very smart exploit, giving it the unprecedented accuracy and speed. Before I started doing this, I had came across a video clip on Wilson's TAFFI demo. The moment he said '...exploits the topology of the image...when you (pinch your fingers), you're creating a new shape...' suddenly gave me a HUGE clue on how he accomplished it.


So basically, you run a threshold on the image. Then do a connected components analysis. You then obtain the component(s) which do not reside on the image borders. You can safely assume that these components are that of the hole created when you pinch your fingers, under the conditions of good thresholding. You can then easily calculate the centroid, the number of pixels, and also the orientation if you first obtain its convex hull (and then get its major axis), or do ellipse fitting.







first image: input image from webcam.
second image: connected component analysis result of the input image. Basically, all pixels which are neighbors to each other are assigned a common label.
third image: convex hull of the shape(s) we find in the connected component analysis. we only perform convex hull on shape(s) not on the boundary of the image.

What's a convex hull, you ask? A convex hull is the smallest convex polygon that fits all data nodes. Convex polygon is a polygon whose inner angles never go beyond 180 degrees. A better definition is all points on a line between any two points in the polygon are always in the polygon, not the outside.

Why do we need convex hull? If you can obtain two points on the convex hull's edge whose line is longest, you can use that, coupled with easy trigonometry to obtain the major angle (minor angle is omitted here). I, however, take the major angle as the orientation of the blob formed by the pinching gesture.

A better way is to use ellipse fitting. This fits an ellipse, instead of a convex hull, on all the data nodes. The major axis of the ellipse can be taken as the orientation.

Alternatively, we could've performed ellipse fitting as well, to stabilize the orientation.





I programmed this in VB.NET. Amazingly, it runs at 13 fps (compiled) on a lowly laptop with Pentium M 730 (2MB L2 cache, 1.6GHz, 533 MHz FSB) and 1.2GB DDR2 RAM. I believe it can be optimized further. If we want to accept gestural input as a means of input, it must reach at least 133 fps. So for now, hardware speed is a problem. Now you see why Minority Report style touch screen or any fancy gestural input never really took off the past few decades.