Microsoft releases new version of Seeing AI mobile app to help low-vision users by describing the world using Azure AI

Microsoft has launched the new Seeing AI app for Android during the March 7th Ability Summit, an event exploring digital solutions for those with disabilities. This innovative app utilizes Azure AI technologies to help people with low-vision abilities navigate life easier.

AI technologies often require powerful computers to analyze and respond to user prompts, and few mobile devices have AI chips such as the Snapdragon 8 Gen 3 in top 2024 smartphones. Therefore, most prompts are passed to online services like OpenAI (ChatGPT) or Microsoft (Azure AI, CoPilot) where more powerful cloud computers handle the requests.

Microsoft Seeing AI utilizes Azure AI for Accessibility to enable responsive, accurate responses on any smartphone with these features:

1. Text-to-speech – The app reads the text it sees. Shorter text can be read in live camera mode while longer text is photographed, converted to text, and finally read aloud.
2. Bar code reader – The app looks up products based on the UPC code.
3. Scene – The app describes the scene it sees, focusing on the nearest object.
4. Person – The app describes the person it sees, identifying the facial emotion.
5. Currency – The app identifies the bills it sees among 17 countries.
6. Color – The app identifies the color in the center of the live photo.
7. Handwriting – The app converts handwritten notes to text.
8. Brightness – The app sounds a tone that varies in pitch depending on the brightness.

The app results are generally accurate, but no AI is as good as a human in identifying and describing what is seen. For example, Seeing AI correctly identifies a shaving razor in a can, but it simply can’t identify a coin sorter despite the cent markings on the body. The same goes with text – the app generally converts text on receipts and packaging accurately, but fails with math and chemical texts.

Although Seeing AI for iOS was launched in 2017 for research into developing new ways to help those with vision impairments, its capabilties were limited because even GPT-1 was not released until 2018. Commercially, Microsoft only began integrating GPT-3 into consumer products in 2021 after the 2020 launch of GPT-3. So Seeing AI was not able to produce the robust results it can today before the latest generative models (such as the 2023 GPT-4 Turbo) were released to the world.

The Microsoft Seeing AI can be downloaded today from the Android app store and Apple app store. Readers who have trouble seeing small things might want to try a lighted magnifier (like this at Amazon).