Google's Project Astra: Gemini app gets eyes

Google shows two new functions for Gemini at this year's MWC in Barcelona. Specifically, it is about Gemini live, i.e. the AI ​​assistant, with whom you can talk about the Android and iOS app in real time. It is updated on Google's latest Gemini model-Gemini 2.0 Flash. This is the version of the multimodal model, which is particularly tailored to the fast, mobile use.

Advertisement

With the update, the Gemini app is able to understand and speak 45 languages. What is new is that you can change the language in the middle of the sentence. To do this, Google announces that one no longer has to change the language settings of the phone-simply continue in a different language and “Gemini Live will understand and answer it”. This function should be available from now on.

Later this month, Google says, the live video entry comes. That is one of the key functions of Project Astra. Google had presented this at the past in -house trade fair I/O. In a video, someone ran through a room with smart glasses and meanwhile kept talking to the AI ​​assistant about what he saw. However, according to Google, Project Astra is a “research prototype for a universal AI assistant”. The live video entry now moves to the smartphone in the form of the app. Gemini also remembers what users have discussed with him, so that you can use it later.

In addition to the video input, there will also be a screen sharing in the future. This makes it possible that you can talk live with Gemini about what can be seen on the phone. Google writes in a press release, so you can buy new jeans in the future.

The visual AI functions will initially only be available for pixel and Samsung devices.

AI assistant and smart glasses

Making AI assistant wiser and more practical is currently the concern of all major AI providers. For example, Openaai has a AI agent called Operator on offer, which can also be said in natural language that he should buy jeans. To do this, he also needs the Advanced Voice Mode. However, when the language mode announced, Openai had also announced that he would get visual skills as Google has now made it available. However, Openaai has not yet published it.

Meta offers a visual AI assistant, especially with its smart glasses, the Ray-Ban Meta Glasses. With these you can look at the surroundings and ask questions directly. Meta AI is responsible for processing – but not yet in the EU.