Emerging AI cyber threats: The next wave of attacks

Introduction

Good afternoon, and thank you for joining today's webinar. The topic is the next wave: emerging AI and cyber threats. I'm Zoe Edmeads, managing director of The Security Company, and I'll be your host. At TSC we have been dedicated to driving behavioural change for over 25 years, and we really understand the significant role people play in building secure cultures. Today's session is presented by our resident cyber security expert, Naz Ali, who will take a deep dive into the new and emerging threats. Please share your questions and thoughts in the chat and Q&A as we go.

Thanks Zoe. My name is Naz Ali, and I'm a resident emerging threats and AI expert here at TSC. What I do is look for the next wave of AI threats that CISOs, DPOs and anyone interested in cyber security should be aware of. Today we're looking specifically at the emerging and AI threats that I don't see enough people talking about, not the usual ones you see in articles, but the more tangential ones everyone should know. With new technology comes new challenges and vulnerabilities that exist outside our foundational security models, things like adversarial attacks, prompt injection and data poisoning. This is a rapidly growing technology used by both defenders and attackers, so it's an area all of us need to understand.

The scale of the threat

Let's start with an overview of the landscape. If cybercrime were a country, it would be the world's third-largest economy, behind only the United States and China. According to Cybercrime Magazine, cyber security costs are set to rise from $8 trillion to $10.5 trillion, with other estimates putting it as high as $15 trillion. We don't yet know how far-reaching the consequences will be, because the technology being used to leverage crime is still innovating and growing.

On the defensive side, there are hundreds of thousands of unfilled cyber security vacancies, not just technical but human too, because there aren't enough experts or enough training to fill the gaps. And the time it takes to create those experts is far longer than the time it takes a threat actor to formulate, run and profit from an attack. In terms of what security leaders think, 58% expect a different set of cyber risks over the next five years, and they rank AI and machine learning as the top theme among the most significant risks.

There's a quote I want everyone to hold onto: cyber threats will be at machine speed, whereas defences are still at human speed. We have endpoint detection that runs faster than humans, but the first stages of most attacks still involve a person spotting, falling for, reporting or dealing with the attack. If attacks come at a speed and scale you simply can't keep up with, what are the consequences?

AI-generated phishing

According to recent Forrester research, 80% of cyber security decision-makers expect AI to increase the scale and speed of attacks, and 66% expect AI to conduct attacks no human could conceive of. Take phishing. Instead of a threat-actor team crafting a campaign, an AI language model can produce one very quickly. What typically took around 16 hours now takes about five minutes and five prompts: who you're targeting, the language, the goal, a call to action and the data you want to extract.

We saw a recent attack where a CEO received a contract by email to review and sign. The language, syntax and design all looked legitimate. The only thing that gave it away was a full stop that landed one space too late, because the email had been generated in a batch of thousands by a language model. The security team caught it, but it almost succeeded. And when AI-generated phishing is combined with vishing and smishing, cloning someone's voice, it becomes far easier to prey on people's worries and insecurities.

Voice cloning and Respeecher

In one example a scammer used samples of a daughter's voice scraped from social media to generate a sentence and trick her father into believing she'd been in an accident. That's the first stage. More frightening is an application called Respeecher, which does this live, in real time, changing a speaker's voice into a selected voice as they talk. It came from Hollywood, used to recreate the voices of actors, but forms of it have been found on the dark web. To build someone's voice you need just 15 words covering the right range of sounds. Any voice can be emulated, which raises serious questions about identity verification whenever we pick up the phone.

Another tool, DeepPhish, uses deep learning to automatically generate phishing emails and messages and then replicate them based on responses. It's trained on previous data sets from the people you're targeting, so a stolen batch of emails lets it craft highly personalised phishing with a far greater chance of success. It learns which email chains to respond to and positions itself as a real person. New versions appear on the dark web every other week.

Deepfakes

Most people know about deepfakes from social media and the proliferation of fake news. We've seen them used to impersonate political leaders, Zelensky, Putin, Biden, Trump, because their images are so readily available. Faces that look completely real can be entirely AI-generated. The next stage is more frightening still: technology like Meta's AR goggles can scan a face and project a photorealistic digital double in real time, so the person you see could be an AI-generated reaction and response.

We've already seen this used to trick executives. A UK energy firm's CEO received a call from his "boss" at the German parent company, complete with the boss's voice and a short video to legitimise it, and transferred $243,000 to a fraudulent account. The cyber insurer paid out, and it changed their protocols. As the digital world becomes this malleable, we may have to start second-guessing everything we see and hear.

AI-powered malware

AI-powered malware has existed since around 2018, but criminals now use "fuzzy models" to develop the next generation. It learns from the environment it's in, updates itself to deal with new security detection, and infects systems without being noticed. AI-powered malware brings four major advantages for attackers: it can hide its code from detection, evade traffic detection, attack the defensive AI trying to stop it, and continuously steal authentication factors from mobile devices.

BlackMamba dynamically modifies its code to slip past automated systems and sense the behaviour of defensive AI; in testing it wasn't detected once. An AI prototype of the famous Emotet trojan runs AI-generated invoice scams, inserting itself into pre-existing email threads so it eventually reaches everyone on the chain. AI talk on the dark web has gone through the roof since ChatGPT launched, with malicious alternatives appearing that can recreate malware strains from research papers or automatically steal common file types from a network.

Passwords, 2FA and DDoS

AI's pattern-recognition ability makes it very effective at cracking weak or reused passwords, often in seconds. Microsoft Threat Intelligence reported a group called Midnight Blizzard that specialises in AI cracking of credentials, running through password and 2FA systems until they break in. They infiltrated over 40 global organisations, including governments, NGOs and tech and media firms, stealing hundreds of credentials and even biometrics and identification documents.

DDoS attacks, long seen as a nuisance, become far more dangerous when supercharged with AI, because they can adapt to your protection in real time. In one recent attack the AI botnet first hit the transport layer, was blocked, then switched to attack the weaker application layer instead, and succeeded.

Emerging threats: data poisoning, public sector and elections

Data and SEO poisoning is something few people are talking about. Attackers use AI to build web pages and content that hit search-engine ranking signals, so a malicious link appears above the legitimate one, without paying for sponsored placement. In one case attackers cloned all of an organisation's high-ranking pages with AI, outranked every legitimate page, and it took weeks to get the malicious links taken down. As a side note, the presenter in the example video was itself entirely AI-generated, a sign of how normal this is becoming.

The public sector is increasingly targeted because defences and awareness often aren't there. We've seen attacks on UK military and defence material, prisons, the Royal Mail and the National Grid, which saw 700,000 malicious emails in just 24 hours. And on democracy, the bigger risk isn't direct hacks on voting machines but social manipulation: fake bots, fake groups and troll farms running hundreds of accounts to influence opinion, with AI generating the content and enabling precise audience targeting based on scraped data.

Quantum attacks

Finally, quantum attacks. There's little evidence yet, but the European Policy Centre has warned of a future "Q-Day", the day a devastating quantum attack happens that no one can deal with, likely within the next five to ten years. Quantum attacks could in theory break any of today's cryptographic algorithms, packaging up an organisation's data and exfiltrating it within seconds. How do you stop an attack that happens in seconds? It will likely come down to quantum defences against quantum attacks, but until we understand the scale and scope, we can't build the defences needed.

Questions and answers

How far behind are the countermeasures, and how quickly are they developed once a threat is identified?

Language models were released to the public, and to threat actors, without regulations or protocols, and we're still catching up. The EU has introduced an AI charter it hopes organisations will adopt. But the strongest countermeasure right now is awareness: many people simply aren't aware of these attacks, and once you are, you naturally build ways to verify things. The big question is whether we'll need to move towards physical verification as digital solutions become harder to trust.

How can we help people stay safe if it only takes 15 words to clone a voice?

It starts with your digital footprint. A lot of people put up information they don't need to, so make profiles private where possible so threat actors can't pull a video with your voice. Your footprint should also reflect your position: if you hold the keys to the castle, you can't be careless. We may also move towards verbal passwords or code words to verify who you're really speaking to. These threats are very recent, and the solutions are still being built; the defensive side is usually slower than the attacking side.

What might attackers do next?

If virtual spaces and the metaverse take off, a frightening possibility is a physical threat caused by a digital change. If a threat actor hacked a headset or virtual room, they could change the digital parameters so that moving in the virtual space leads you into real-world danger, walking into a wall, for example. It's a cyber threat that could cause real physical harm.