Apple has introduced a new version of its Ferret AI model that could move Siri closer to seeing and controlling iPhone apps. The updated system, called Ferret-UI Lite, focuses on running directly on a device rather than relying heavily on cloud servers.
This shift matters because Apple has long promoted privacy and local processing. Instead of sending user data to large remote servers, the company aims to handle more tasks on the iPhone itself.
From Research Project to Practical Tool
Apple first revealed Ferret in 2023 as a multimodal large language model developed with Cornell University.The system could analyze selected regions of images and respond to questions about them. Later, the team expanded the idea into Ferret-UI, which could read and interpret elements within a phone’s interface.
However, earlier versions depended on large cloud-based models. While powerful, those systems required external processing. As a result, they were not ideal for private, real-time smartphone tasks.
Ferret-UI Lite changes that approach. It uses a smaller 3-billion-parameter model trained on real and synthetic interface data.Therefore, it can run more efficiently on mobile hardware.
How Ferret-UI Lite Works
The new model uses chain-of-thought reasoning and reinforcement learning to improve decisions. In addition, it introduces a zoom-in mechanism. As shown in the diagram on page 3, the system first predicts where important elements appear on a screen. Then it crops the image around that area to analyze it more closely.
Because it focuses on smaller regions, it processes less data. Consequently, it can refine results faster. Researchers say this approach mirrors how humans look closely at details.
Performance and Limitations
In benchmark testing, Ferret-UI Lite achieved 53.3% accuracy in the ScreenSpot-Pro GUI grounding test. Notably, it outperformed a larger 7-billion-parameter model by more than 15% in that test. Still, it showed weaker results in some navigation tasks.
Overall, the research highlights both promise and challenges. While the model is not revolutionary, it proves that smaller AI systems can handle complex interface tasks locally.
If Apple continues this path, Siri may soon interact with apps more intelligently—without sacrificing speed or privacy.
