Microsoft is testing a Copilot Vision update that can see your screen

Microsoft has started testing a new update for its Windows Copilot Vision app, allowing users to share a screen or certain apps with the AI assistant.

According to The Verge’s Tom Warren, Copilot Vision can perform tasks like coaching users through Adobe Photoshop features or analyzing webpages or photos that the user is looking at. Warren, who got a sneak peek of Copilot Vision AI at Microsoft’s 50th Anniversary party, noted that the assistant guided him through Minecraft and helped him optimize the settings of the video editor Clipchamp.

Copilot will also be able to highlight parts of the screen to assist users through different apps. However, this feature will not be enabled in the initial release yet.

Testing has also been done for a file-searching feature for the Copilot, allowing users to ask the AI assistant about the contents of a file on the user’s PC.  Currently, it supports .docx, .xlsx, .pptx, .txt, .pdf, and .json file types. Conveniently, they are all Microsoft-based file types, although they will most likely expand it to include more file types.

This feature would be most convenient for professionals who need to search a file they were working on quickly or use it to identify a file that may have been misnamed. These new features will only require the Copilot app on Windows and will be available on iOS and Android versions.

Currently, Microsoft is testing with Windows Insiders and has limited testing for Copilot Vision to U.S. testers only. The new feature will roll out in the coming months with the latest Windows 11 update.

Source: The Verge