Cybercriminals are like everyone else leveraging AI-powered tools themselves, not just to generate attacks, but to weaponise the recipient’s own inbox AI features! A newly disclosed vulnerability in Gmail’s Gemini summarization feature raises fresh concerns for the email security landscape.
A security researcher has revealed a novel form of email phishing that bypasses traditional detection mechanisms by exploiting Google Workspace’s AI-powered summarizer, Gemini. First reported by BleepingComputer, the attack vector requires no links, no attachments, and no visible malicious content – instead, it relies on prompt injection, hidden invisibly in the body of the email.
By embedding invisible prompts using techniques such as hidden HTML/CSS styling, attackers can manipulate the Gemini summary that appears in the Gmail interface. In a demonstrated case, the summary falsely warned:
“Gemini has detected your Gmail password has been compromised, please call us immediately at [phone number].”
This carefully crafted hallucination is a classic social engineering trick—designed to trigger panic and an immediate response, in this case a phone call to a scammer. Because the original message contains no visible threats, links, or traditional phishing payloads, it can pass through spam and phishing filters and land directly in the user’s inbox, where Gemini’s summarizer unwittingly delivers the attack on the attacker’s behalf.
A New Class of AI-Driven Threats
Google has acknowledged the issue and stated that it is “hardening [its] protections against prompt injection attacks,” though it claims to have found no evidence of this exploit being used at scale in the wild.
Nonetheless, this incident underscores a growing class of email threats unique to AI-infused interfaces. As email clients integrate generative tools for summarization, prioritization, and even sentiment analysis, attackers are beginning to design exploits that target how machines interpret email content, rather than how humans do.
Why It Matters
For security teams, ESPs, and email platforms, this development signals a shift: it’s no longer enough to validate what a user sees or clicks. AI-generated UI elements—summaries, alerts, previews—are now attack surfaces in their own right.
This case highlights the need for:
- Improved input sanitization in AI summarization engines.
- New heuristics to detect prompt injection attempts.
- User education about relying on AI-generated summaries, especially when they relate to security or account status.
For legitimate senders, this is a reminder to maintain clarity and consistency in message formatting, and to monitor inbox renderings, especially in platforms deploying AI-generated content overlays.
Looking Ahead
As generative AI becomes more embedded in email platforms, so too will the risks of manipulation. The Gemini case may be the first widely reported exploit of its kind, but it likely won’t be the last. In fact Google acknowledged the potential for abuse exactly one month before prompt injection abuse was reported as being successful.
