The new tools, called Camera Switches and Project Activate, use an Android phone's front-facing camera and machine learning to detect face and eye gestures. They effectively turn a front-facing camera into a switch -- an adaptive tool that replaces a keyboard, mouse, or touchscreen functions.
Camera Switches is a feature within the Android Accessibility suite that lets users navigate their phones with eye movements and facial gestures. For instance, a user could look left to move across items on the screen or raise their eyebrows to select an item. Users can customize the feature to fit their range of movement. It's available globally and in 80 languages.
Meanwhile, Project Activate is a new app that lets users activate customized, preset communications by making facial gestures, such as smiling or looking up. A user can make a gesture to play a specific text-to-speech phrase, send a specific text message, make a phone call, play audio or control a smart speaker. While the app lets users express short phrases like, "please wait," it's not meant to replace a speech-generating device.
Google also expanded the Lookout app, which uses computer vision to help people with impaired vision. Now, the app can read out handwritten text in Latin-based languages, so users can read notes or letters sent to them. Additionally, Lookout's currency mode was expanded to include rupees and euros.