Abstract: Traditional smartphone interactions such as touch-typing and voice are not suited for every situation. For example, studies have shown that blind users find typical smartphone touch interactions challenging. Similarly, voice interactions are not always convenient due to privacy concerns and due to noise-levels in outdoor environments. On the other hand, there is a rise in wearable technology including smartwatches and earables that are equipped with sensors. Our work investigates if wearable sensors can be used as alternate modes of interaction for smartphones.

In this talk, I will first present AccessWear, a system designed for accessible smartphone touchscreen interaction for blind users. AccessWear allows a blind user to use smartwatch gestures to replace smartphone touch screen gestures. We propose a lightweight gesture recognition system that works well specifically for blind users, requires no personalization, and can run on resource constrained devices. Further, we propose a input virtualization technique that can replace a touchscreen input with an alternate gesture without requiring changes to the application. Our study with 8 blind users finds that we can recognize gestures with a 92% accuracy and end-to-end latency when using an alternate gesture is less than 100ms.

In the second part of the talk, I will describe JawSense, a hands-free and privacy-preserving technique which allows user to interact with smart devices just by moving their jaw. It senses the muscle deformation and vibration caused by unvoiced speaking to decode the unvoiced phonemes spoken by the user even in presence of body movements and external noise. Our evaluation with six subjects shows that JawSense can be used as a hands-free and privacy-preserving interaction technique with 92% phoneme classification rate.