Notebookcheck Logo

Microsoft releases new version of Seeing AI mobile app to help low-vision users by describing the world using Azure AI

Microsoft Seeing AI leverages Azure AI tech to help people with low-vision to hear the world around them described. (Source: Microsoft)
Microsoft Seeing AI leverages Azure AI tech to help people with low-vision to hear the world around them described. (Source: Microsoft)
Microsoft has released a new version of its Seeing AI mobile app for Android devices during the Microsoft Ability Summit. This speaking app helps people with poor vision to see the world around them by reading text, describing scenes, and identifying people, objects, colors, and currencies using Azure AI technologies.

Microsoft has launched the new Seeing AI app for Android during the March 7th Ability Summit, an event exploring digital solutions for those with disabilities. This innovative app utilizes Azure AI technologies to help people with low-vision abilities navigate life easier.

AI technologies often require powerful computers to analyze and respond to user prompts, and few mobile devices have AI chips such as the Snapdragon 8 Gen 3 in top 2024 smartphones. Therefore, most prompts are passed to online services like OpenAI (ChatGPT) or Microsoft (Azure AI, CoPilot) where more powerful cloud computers handle the requests.

Microsoft Seeing AI utilizes Azure AI for Accessibility to enable responsive, accurate responses on any smartphone with these features:

  1. Text-to-speech – The app reads the text it sees. Shorter text can be read in live camera mode while longer text is photographed, converted to text, and finally read aloud.
  2. Bar code reader – The app looks up products based on the UPC code.
  3. Scene – The app describes the scene it sees, focusing on the nearest object.
  4. Person – The app describes the person it sees, identifying the facial emotion.
  5. Currency – The app identifies the bills it sees among 17 countries.
  6. Color – The app identifies the color in the center of the live photo.
  7. Handwriting – The app converts handwritten notes to text.
  8. Brightness – The app sounds a tone that varies in pitch depending on the brightness.

The app results are generally accurate, but no AI is as good as a human in identifying and describing what is seen. For example, Seeing AI correctly identifies a shaving razor in a can, but it simply can’t identify a coin sorter despite the cent markings on the body. The same goes with text – the app generally converts text on receipts and packaging accurately, but fails with math and chemical texts.

Although Seeing AI for iOS was launched in 2017 for research into developing new ways to help those with vision impairments, its capabilties were limited because even GPT-1 was not released until 2018. Commercially, Microsoft only began integrating GPT-3 into consumer products in 2021 after the 2020 launch of GPT-3. So Seeing AI was not able to produce the robust results it can today before the latest generative models (such as the 2023 GPT-4 Turbo) were released to the world.

The Microsoft Seeing AI can be downloaded today from the Android app store and Apple app store. Readers who have trouble seeing small things might want to try a lighted magnifier (like this at Amazon).

The Seeing AI app is generally very good at recognizing common text. (Source: Notebookcheck)
The Seeing AI app is generally very good at recognizing common text. (Source: Notebookcheck)
The app can get confused in its description if the image shifts slightly. (Source: Notebookcheck)
The app can get confused in its description if the image shifts slightly. (Source: Notebookcheck)
Common objects are easily recognized, but items like a coin sorter confuse the AI. (Source: Notebookcheck)
Common objects are easily recognized, but items like a coin sorter confuse the AI. (Source: Notebookcheck)
Although both are well-known leaders, Seeing AI gets one age wrong. (Source: Notebookcheck)
Although both are well-known leaders, Seeing AI gets one age wrong. (Source: Notebookcheck)
Seeing AI is very good at identifying stuffed toys, but gets confused if a plastic garment bag is next to one. (Source: Notebookcheck)
Seeing AI is very good at identifying stuffed toys, but gets confused if a plastic garment bag is next to one. (Source: Notebookcheck)
Read all 1 comments / answer
static version load dynamic
Loading Comments
Comment on this article
Please share our article, every link counts!
> Expert Reviews and News on Laptops, Smartphones and Tech Innovations > News > News Archive > Newsarchive 2024 03 > Microsoft releases new version of Seeing AI mobile app to help low-vision users by describing the world using Azure AI
David Chien, 2024-03- 8 (Update: 2024-05-28)