Even before releasing iOS 18.1 to the public, Apple has released the first developer beta of iOS 18.2, which includes many new Apple Intelligence features. One of the new features is the long-awaited integration with ChatGPT, and thanks to this, you can now ask Siri for details about what you’re seeing on your device’s screen.

Use Siri to ask ChatGPT what you’re seeing on your screen

With ChatGPT integrated into Siri, Apple’s virtual assistant will use the OpenAI platform to answer complex questions. However, it can also read what’s on the screen of your iPhone, iPad or Mac. For example, if you’re watching a video or looking at someone’s photo on Instagram, you can ask things like “where is this?” or “what building is in that photo?”

Siri will then take a screenshot and upload it to ChatGPT, which is able to understand the context and use its database to provide an answer. Of course, for privacy reasons, Siri always asks for permission before capturing a screenshot of your device and sending it to ChatGPT.

The feature is particularly interesting for those who don’t have an iPhone 16, which is the only device so far to get Visual Intelligence. With this feature, users can press Camera Control to get information about what they’re seeing on the camera. Visual Intelligence uses both Apple Intelligence, ChatGPT, and Google to provide answers.

If you’re on an iPhone 15 Pro or a compatible iPad, you can simply take a photo and then ask Siri and ChatGPT whatever you want.

It’s worth noting that Apple is also working on its own onscreen awareness feature using Apple Intelligence language models. However, this feature will come with another iOS 18 update – probably next year. Once this becomes available, Siri will understand the context and take actions based on what’s on your screen – all processed on-device.

More about iOS 18.2

In addition to ChatGPT integration, iOS 18.2 also comes with Image Playground, Genmoji and the redesigned Mail app.

For now, iOS 18.2 beta is only available for devices compatible with Apple Intelligence. That means iPhone 15 Pro or later and M1 iPads or later. Of course, the update will eventually be available for other devices too. Apple previously suggested that iOS 18.2 will be released to the public in December.

iOS 18.1 will be released next week with Apple Intelligence Writing Tools and Summarization of notifications and text.

Source: 9to5mac