Apple is taking a major step toward digital accessibility with its latest AI project: an assistant that can describe Street View scenes to people who are blind or visually impaired. The new tool, called SceneScout, could help users explore places virtually before visiting them.
What SceneScout Does
SceneScout uses a large language model trained to interpret and describe Street View imagery. It can explain visual elements such as trees, buildings, and paths. The goal is to provide blind users with a better sense of an area’s layout before they travel there.
This tool builds on existing navigation aids but offers something new. Current solutions like Microsoft’s Soundscape only work in real time, while SceneScout allows for advance exploration. It includes two main modes:
- Route Preview, which outlines key details along a route.
- Virtual Exploration, which lets users move through scenes freely and hear what’s around them.
Promising Test Results
In early tests, users found SceneScout helpful. The system gave accurate descriptions about 72% of the time, and it could identify stable visual elements with up to 95% accuracy. Although minor errors still exist, the feedback suggests that SceneScout fills an important gap.
Future Possibilities
Test participants also shared ideas to improve the tool. Some suggested personalized descriptions based on user preferences. Others recommended shifting the AI’s point of view to match a pedestrian’s height, not a car-mounted camera.
Apple may eventually pair this AI with future products. For instance, AirPods with cameras or Apple smart glasses could feed real-time visuals to the AI. This would allow users to receive live descriptions of their surroundings as they walk.
