Touch+Finger: Extending touch-based user interface capabilities with “idle” finger gestures in the air

In this paper, we present Touch+Finger, a new interaction technique that augments touch input with multi-finger gestures for rich and expressive interaction. The main idea is that while one finger is engaged in a touch event, a user can leverage the remaining fingers, the “idle” fingers, to perform a variety of hand poses or in-air gestures to extend touch-based user interface capabilities. To fully understand the use of these idle fingers, we constructed a design space based on conventional touch gestures (i.e., single- and multi-touch gestures) and interaction period (i.e., before and during touch). Considering the design space, we investigated the possible movement of the idle fingers and developed a total of 20 Touch+Finger gestures. Using ring-like devices to track the motion of the idle fingers in the air, we evaluated the Touch+Finger gestures on both recognition accuracy and ease of use. They were classified with a recognition accuracy of over 99% and received positive and negative comments from 8 participants. We suggested 8 interaction techniques with Touch+Finger gestures that demonstrate extended touch-based user interface capabilities.