Microsoft Empowers AI to Interpret Computer Screens

Microsoft has enhanced its AI assistant, Copilot, to comprehend contextual information from computer screens during internet browsing.

When users activate Copilot Vision, the neural network perceives web pages, understanding both text and images. Users can then inquire about the websites they visit using Microsoft Edge. For instance, upon viewing a dish on the screen, the AI can provide its recipe.

Other features of Copilot Vision include:

• translation and summarization of text;
• performing tasks such as showcasing discounted products in a store catalog;
• assisting in games like chess.

“This is a new way to invite artificial intelligence to navigate the internet with you, conveniently positioned at the bottom of the Edge browser when assistance is needed,” Microsoft noted.

The corporation promises to delete user data after each session, ensuring that the information gathered by the AI is not used for training purposes.

In October, Microsoft introduced an updated version of the Copilot AI assistant with voice mode and screen image recognition.