Ghost Fingers: Intuitive Keyboard and Multi-Touch Interaction
Project developed at: HP/Palm
The "Ghost Fingers" project introduced a novel interaction method for seamless switching between keyboard input and multi-touch gestures on displays beyond arm's reach. This system provided a translucent, real-time visualization of the user’s fingers and hands over the remote display, enabling precise and intuitive direct manipulation through a closed interaction loop.
One prototype featured a wireless keyboard equipped with a micro camera facing the keys. The camera served two purposes: tracking the user’s hand and finger positions and generating a live, translucent overlay of the fingers on the remote UI. This approach allowed users to maintain a natural and efficient workflow, bridging the gap between keyboard and touch interactions.
The prototypes delivered excellent results, offering an impressive user experience. The innovative elements were patented in two separate applications, both of which were granted, and the system was documented in a published research paper. While the patents were later sold to Qualcomm as part of a larger IP transfer, they were eventually abandoned by Qualcomm due to a business decision, making this IP publicly available for future use by forward-thinking companies interested in bringing this concept to life.
This image illustrates the Ghost Fingers prototype: a standard QWERTY keyboard equipped with an image sensor, such as a webcam, to track the position of one or both hands over the keys. In text-input mode (left), the keyboard functions normally. By pressing a designated key (e.g., CTRL), the system switches to "multi-touch mode" (right), enabling finger tracking.
In multi-touch mode, a transparent, real-time overlay of the user’s hands and fingers is displayed on the remote screen, allowing for intuitive interaction with the UI. Users can manipulate elements on the screen, such as pressing icons, without needing to touch the display directly, creating a seamless blend of keyboard and touch input.
Company: HP/Palm R&D Center
Team: Future Concepts and Prototyping team
Location: Sunnyvale, CA
Year: 2011
This video showcases a working prototype of the Ghost Fingers system, illustrating a novel interaction method for seamlessly switching between keyboard input and multi-touch gestures on out-of-reach displays. While the display in this demonstration is within arm's reach, the intended application is for distant displays, such as TVs or large monitors.
This figure demonstrates simultaneous text and multi-touch input using the Ghost Fingers system. The left hand operates in multi-touch mode, manipulating on-screen elements like cards, while the right hand remains in text-input mode, typing into an email message window. Only the left hand is visualized on the GUI, as its representation aids interaction, whereas the right hand is not displayed since visualization is unnecessary for typing.
One of the images from the granted patent