Why AI-Written Phishing Is Outpacing the Native Filters Built to Stop It

The Scale of the Problem

Email remains the most common way into an organisation, and the shape of that risk has changed faster than most defences have. According to threat intelligence firm SOC Radar, phishing volume has climbed more than 4,000% since ChatGPT became a household name. The number sounds abstract until you consider what it represents: the tools available to an attacker today write more convincingly, sound more human, and cost next to nothing to run.

The financial picture backs this up, and the pattern is global rather than confined to any one country. Interpol's 2026 Global Financial Fraud Threat Assessment puts worldwide fraud losses at around $442 billion in 2025, and finds that schemes using AI are 4.5 times more profitable than traditional fraud, a gap the agency expects to widen over the next three to five years as the technology gets cheaper and easier to deploy. Most fraud reporting, Interpol's included, does not yet break out how many of those cases involved AI specifically, which means the true share is almost certainly higher than any current estimate captures.

A Case in Point

One case shows what this looks like in practice. In January 2024, a finance employee at the British engineering firm Arup received an email that appeared to come from the company's UK-based chief financial officer, requesting a confidential transaction. Wary of a scam, he asked for a video call to verify it, and got one –– a conference with the CFO and several colleagues he recognised, all discussing the transfer. Every person on that call was an AI-generated deepfake, built from publicly available audio and video of the real executives. Believing the request was genuine, the employee authorised fifteen transfers totalling roughly $25 million before the fraud was discovered. It remains one of the largest deepfake-enabled frauds on record, and it began the same way most email attacks do: with an urgent message dressed up as coming from someone the victim trusted.

Academic research backs up what that case suggests anecdotally. In a controlled 101-person trial, Harvard researchers found that AI-generated phishing emails achieved the same 54% click-through rate as messages written by professional human attackers, at a cost of roughly four cents per email in API fees, compared with around 30 minutes of a human attacker's time spent drafting each message. Microsoft's 2025 Digital Defence Report reaches a related conclusion at a larger scale: people are 4.5 times more likely to click on AI-assisted phishing than on manually written phishing, a gap Microsoft says can make campaigns up to fifty times more profitable once engagement and automation efficiency are factored in. 

Separate fieldwork from the University of California, Berkeley's Centre for Long-Term Cybersecurity reaches a related conclusion from a different angle: attackers have largely abandoned mass-blast scams in favour of campaigns built on AI-assisted research into each specific target, the same kind of preparation that made the Arup call believable in the first place. As Dr Gil Baram of Bar Ilan University and the Berkeley centre puts it, generative AI is helping cybercriminals outpace traditional defences.

Why Native Filters Fall Short

None of this is an argument against native email filtering. Google Workspace and Microsoft 365 have both added machine-learning-based impersonation and behavioural detection to their filtering over the past several years –– and against known or previously seen threats, that combination still performs well. 

The trouble is that those models are trained predominantly on prior examples of bad behaviour, and today's most damaging attacks are built specifically to have no precedent to learn from: no known signature, no blocklisted domain, no prior campaign to pattern-match against. That is precisely why phishing and stolen credentials remain among the handful of ways attackers most commonly get in, year after year, across major industry breach reports, despite sustained investment in filtering technology, native and third-party alike, over the same period.

What is landing in inboxes now tends to fall into three categories.

• The first rides on trust rather than technical trickery: a credential-harvesting page hosted on a lookalike Microsoft 365 or Google Workspace login screen, or a DocuSign or SharePoint notification generated fresh for each target rather than reused from a template, so there is nothing for a reputation-based filter to fingerprint.
   
• The second dispenses with links and attachments altogether. Attackers now pair large language models with information scraped from LinkedIn, company websites and press coverage to write a message tailored to one person and one relationship, sometimes injecting a convincing reply into a live email thread, a technique researchers call conversation hijacking, to request a wire transfer. There is nothing for a filter to scan, because the entire attack is the sentence itself.
   
• The third changes channel entirely: callback phishing, what researchers increasingly refer to as TOAD, or telephone-oriented attack delivery, moves the con to a phone call; QR codes hide a malicious link inside an image no text scanner can read; calendar invitations auto-accept before anyone has read them.

What It Costs

The cost of getting this wrong is considerable. IBM's 2025 Cost of a Data Breach Report puts the average phishing-caused breach at $4.8 million, above the $4.44 million global average across all breach types. Part of the reason is speed: the same report found generative AI has cut the time it takes to write a convincing phishing email from roughly sixteen hours, start to finish, including the research and targeting that goes into it, to about five minutes. The economics have shifted in the attacker's favour faster than most organisations have adjusted their defences.

A Different Kind of Defence

The response this calls for is not simply more filtering of the same kind. A category of tools that analysts, including Gartner, refer to as Integrated Cloud Email Security, or ICES, sits alongside native filtering rather than replacing it, reasoning about what a message is actually asking someone to do rather than whether it resembles something seen before. The stronger implementations pair that detection with training that reaches an employee within days of a near miss, not at the next annual refresher. There is a fitting symmetry to the research behind this shift: the same Harvard team that tested how well AI could write phishing emails also tested whether AI could catch them, and found that a large language model correctly flagged malicious intent in roughly 97% of cases with zero false positives, including messages that had already slipped past human reviewers. If artificial intelligence can write a phishing email a person will not catch, there is good reason to believe the same technology, applied to defence, can catch what a person would otherwise miss.

Understanding what your current email security misses starts with examining the threats that still reach employees. Compare the phishing messages your filters catch with those employees report, and track how that balance changes over time. An increase in employee-reported phishing warrants investigation: it may reflect gaps in detection, improved reporting, or both.

Look closely at the types of attacks getting through, particularly reply-chain hijacking, calendar-invite lures, QR codes, and urgent wire transfer requests. These patterns can help reveal where your existing defences need additional support.

For a closer look at the threats reaching your inboxes, Adaptive Security's Enterprise AI Email Security Scan can help identify attacks that have passed through your existing filters.