Developed by Microsoft, ‘Seeing AI’ is a free iPhone app designed to aid low-vision individuals in understanding and navigating the environment around them. The app’s software integrates use of the built-in camera to recognize people, specific items, or a general environment around the user. This technology would be particularly useful within art museums, where ‘Seeing AI’ users could gain a better understanding of a painting or sculpture, for example.
Personal iPhone Screenshot of the ‘Seeing AI’ home screen — Source: the author
When the user opens the app, they select a category that matches what they want described to them. Categories include: Short Text, Document, Product Barcode (scan barcodes), Person (recognize faces, read facial expressions), Currency, Scene, Color (describe color), Handwriting, and Light (describe light). Once the proper category is selected, the user points their phone at the item, person, or scene. The app then replies with a description of what the camera captures.
Personal iPhone screenshot of an accurate description of a scene — Source: the author
The benefits of this tool are many. The app is moderately accurate, working 7/10 times for the author, and the categories cover many items or scenes the individual may want read or described to them. A meaningful feature of this app is that it can recognize and remember faces. This could allow low-vision individuals to more quickly find friends and family in crowded locations (e.g. in a museum or arts festival). Finally, the app can be paused (and un-paused) at any point. This function allows users to select the correct category without the distraction of the app describing what the camera captures.
Personal iPhone screenshot of an incorrect description of nearly the same scene — Source: the author
Unfortunately, the app is not always reliable. When reading text, for example, the app occasionally misses words—perhaps due to the color of the text and/or background. Additionally, the color function of ‘Seeing AI’ is incorrect more often than it is correct. When the app makes a mistake, it does not always identify uncertainty. For example, the “Scene” category is the only one in which ‘Seeing AI’ includes the modifier “probably.” Quite possibly the biggest issue with this app is that the text in certain menus is neither read to the user nor in large font. While ‘Seeing AI’ tells the user which category they have selected, functions such as the pause and play buttons may not be visible. This could pose an enormous problem if the user has difficulty navigating the app or wants to change settings, and it may require outside assistance because the app cannot be used with voice commands.
Personal iPhone screenshot of ‘Help’ section — Source: the
‘VisionAssist,’ a similar app available for download for $1.99, acts as a text magnifier for those with low-vision conditions such as age-related macular degeneration, glaucoma, diabetic retinopathy, cataracts, and color blindness. While this app is limited to text and does not read text aloud, it is an alternative that some members of the low-vision community could find more useful.
Though ‘Seeing AI’ is an exciting example of how technology can make the world at large accessible for everyone, it falls short of its goal because the entire app should be easily accessible by vision-impaired individuals, not just part of it. Independence is important. For certain individuals in the low-vision community, this app will work very well, but for those who are very limited by vision, other technologies are emerging on the marketplace that may provide different but overlapping functionalities.
Sources:
Microsoft. “Seeing AI app.” Downloaded and accessed September 10, 2018.
Pivothead. “Seeing AI Project.” Accessed September 15, 2018.
http://www.pivothead.com/seeingai/.
Pivothead. “The SMART Series.” Accessed September 15, 2018. http://www.pivothead.com/.