Microsoft Enhances Copilot with Screen Analysis and App Assistance

Microsoft has commenced testing an update for Copilot on Windows, enabling the AI to analyze screen images and provide guidance on various application functionalities, according to The Verge.

Copilot Vision will assist with basic functions in Adobe Photoshop or analyze photos and visited web pages. In one example, the AI assistant aided with Minecraft gameplay and optimized settings in the video editor Clipchamp.

Copilot can highlight parts of the screen to guide users while working with applications.

“Copilot Vision might seem similar to Microsoft’s Recall feature, which automatically takes screenshots if permitted, but it is more akin to sharing an app screen or the entire desktop in Microsoft Teams,” the publication emphasized.

Microsoft has also begun testing a file search feature in Copilot on Windows, such as searching by document text.

Both features require only the Copilot app on Windows, not a full Copilot Plus computer. The test began in the Windows Insiders program, with deployment to Windows 11 users expected in the coming weeks or months.

In October 2024, Microsoft introduced an updated version of the Copilot AI assistant with voice mode and screen image understanding.

Later, the company announced the addition of two “reasoning” AI agents to Microsoft 365 Copilot for analyzing large volumes of information and providing expert insights.

Back in March, Xbox unveiled Copilot for Gaming—an AI-driven assistant designed to be a “personal gaming companion.”