Hackers could exploit this Gemini mail summaries loophole for phishing attacks
Google added Gemini enabled mail summaries in Gmail around end of May to help users get a concise gist without having to comb through paragraphs of text. However, a Gemini flaw could allow hackers to launch a prompt-injection phishing attack on a user, especially those who have become reliant on AI summaries for their email workflows.
Identified by researcher Marco Figueroa, the GenAI Bug Bounty Programs Manager at Mozilla, the problematic email would look like any other random email full of text but could be hiding a phishing scam that Gemini would fail to identify. The malicious instructions can be hidden within the body of the text or right after buy changing them to 0 font size and white color, making them invisible. However, Gemini would still parse that part of the mail and follow whatever instructions it holds.
As an example, Figueroa hid a warning message for Gemini about the user’s Gmail password being compromised along with a support phone number. Upon summarizing it, the AI showed the warning at the end and the call to action for immediately calling the support phone number. While not everyone would be fooled by this message, some might follow through out of fear of their account being compromised.
The researcher adds that security teams can implement detection and mitigation methods for content that has been formatted to be hidden, in order to then remove or ignore the content. There could also be post-processing filters to look at what Gemini is outputting and then identify URLs, urgent messages, or phone numbers.
BleepingComputer reached out to Google regarding this Gemini loophole, to which a representative shared that some of the mitigation strategies are in the process of being implemented.
Source(s)
