t h e   v i s u a l   t o u c h p a d
a    t w o – h a n d e d    g e s t u r a l    i n p u t    d e v i c e
Shahzad Malik, Joe Laszlo
Department of Computer Science
University of Toronto
hand segmentation
Using background subtraction, the hand regions and contours can be extracted from the image.  Strong peaks in the contours are then marked as fingers, with a heuristic approach to decide on finger labels.  Finger orientation is determined by fitting a line through the fingertip and the midpoints along the finger region.  Postures and gestures can then be recognized by using both static and temporal finger information.
t e c h n o l o g y
touchpad detection
Assuming the touchpad is a simple piece of paper with a black rectangle defining the active area, image thresholding is used to extract the touchpad from an image.  Using a homography, the corners of the touchpad are then mapped to the corners of the display.  
Acknowledgements Ravin Balakrishnan, Allan Jepson, Abhishek Ranjan
ICMI 2004, State College, PA, October 14-15, 2004
o v e r v i e w
a low-cost, multi-point, touch-sensitive device
By using two off-the-shelf web cameras and simple computer vision algorithms, the Visual Touchpad provides a low-cost alternative to more expensive touch-sensitive hardware.  The system can detect fingertip positions for all five fingers of each hand, as well as the 2D orientation of each finger.
layered gesture recognition
Unlike standard touch-sensitive surfaces, the Visual Touchpad allows hand gestures to be recognized both on the touch surface, as well as above it, allowing for a wider range of gestures and interactions.
“hands-up” display
By segmenting the hands from the live video streams, they can be rendered directly on the screen.  This allows for a compelling direct manipulation interface that compares favorably to touch-sensitive screens or surfaces, but without the occlusion or fatigue problems associated with such devices.
single device for text-entry and navigation
With the ability to detect all five fingers of each hand, the Visual Touchpad can be used for natural text entry with a customized, on-screen virtual keyboard.  This can potentially replace both the keyboard and mouse with a single interface, which is useful for applications that require a user to frequently alternate between text-entry and mouse navigation modes.