For twenty years, phishing defense leaned on a quiet assumption: the attacker would make a mistake you could see. Bad grammar, a stiff translation, a generic greeting, a logo stretched slightly wrong. We built entire awareness programs around teaching people to notice the seams. Generative AI removed the seams. The economics that used to force attackers to choose between volume and quality no longer apply — they can now have both, and the tells your staff were trained to spot have largely stopped appearing. This is not a future problem to plan for. It is the current baseline of what lands in your inboxes and rings your phones.
The Constraint That Disappeared
Convincing social engineering was always possible; it was just expensive. A tailored pretext against a specific finance manager required an operator to research the target, understand the org chart, learn the internal vocabulary, and write something that sounded native. That effort meant spear phishing was reserved for high-value targets, and everyone else got the sloppy mass-mailed version that awareness training was designed to catch.
Language models collapsed that cost to near zero. The same effort that once produced one good lure now produces thousands, each personalized to a specific person, in fluent idiomatic language, matching the tone of the sender being impersonated. Real-time translation means a crew with no local language skills can operate credibly in any market you do business in. The mass-mail tier and the spear phishing tier have merged, and the merged tier looks like the expensive one.
Reconnaissance at Machine Speed
The lure is only half of it. The other half is knowing enough about the target to be plausible, and that intelligence is sitting in public view:
- Organizational mapping: job postings, professional networks, and conference talks reveal reporting lines, tooling, and who has authority to move money or grant access.
- Tone and vocabulary capture: public writing, recorded talks, and social posts give an attacker enough material to imitate how a specific executive actually writes and speaks.
- Timing intelligence: earnings cycles, product launches, travel posts, and out-of-office signals tell an attacker when a request will feel routine and when the person who could verify it is unreachable.
- Technical fingerprinting: vendor case studies, support forum posts, and job requirements expose the exact SaaS stack a pretext should reference to sound internal.
Collecting and correlating this used to be days of analyst work. Automated, it is minutes — and it feeds directly into the lure, which is why the message referencing your actual vendor, your actual project name, and your actual approval workflow no longer implies an insider.
Deepfakes Move the Attack Off Email
Voice and video synthesis has crossed the threshold where a short sample of public audio produces a convincing clone. That changes vishing from a low-yield tactic into a serious one, and it breaks a control many organizations rely on without realizing it: the callback. "I'll verify by phoning them" and "I'll ask them to hop on a video call" both assume the voice and face are proof of identity. They are not, and treating them as proof is now an exploitable gap.
The pattern to expect is multi-channel. An email establishes a pretext. A voice call that sounds like a known executive adds urgency and authority. A video presence closes the deal. Each channel corroborates the others, and the target's confidence rises with every touchpoint — which is precisely what makes the sequence so effective. Urgency, authority, and confidentiality remain the core levers; AI just makes them arrive in a package your people were never trained to doubt.
Stop Training Eyes, Start Fixing Processes
If the message is indistinguishable from legitimate communication, detection by inspection is a dead end. Telling people to look harder at something that looks correct produces anxiety, not security. The defense has to move from perception to process.
That means out-of-band verification for any consequential action — payment changes, credential resets, access grants, data extracts — using a channel and contact detail established in advance, never one supplied in the request itself. It means codewords or internal challenge-response for high-value voice and video requests, because the medium no longer authenticates the person. It means dual authorization on financial and access changes so no single deceived employee is sufficient. It means least privilege, so a successful pretext yields limited reach. And it means making it socially safe to slow down and verify a request from an executive, because most of these attacks succeed by exploiting the reluctance to question authority under time pressure. Technical controls still matter — strong phishing-resistant MFA in particular — but they work by removing the thing the attacker is asking for, not by helping people spot the ask.
Your AI Systems Are Targets Too
There is a second front here. The AI you deploy — support agents, copilots, retrieval systems, anything wired to tools or data — can itself be socially engineered. Prompt injection is social engineering aimed at a model that has no skepticism, no escalation instinct, and often more privilege than the humans it serves. Content that reaches your model from a ticket, a document, or a web page can carry instructions, and a model with tool access can be talked into acting on them. Rarefied's AI red teaming work targets exactly this, treating your models and agents as attack surface rather than product features.
Closing
Testing is the only honest way to know where you stand. Not a canned phishing simulation with a deliberate typo, but adversary simulation that uses real reconnaissance, credible pretexts, and multi-channel pressure against your actual verification processes — the approach described in our methodology. The result tells you whether your controls hold when the lure is perfect. Contact us to find out before someone else runs the test for you.