In late 2022, AI exploded into the mainstream with OpenAI’s ChatGPT, starting an AI-fuelled shift in both everyday life and the cyber threat landscape. Just as quickly as everyday users rushed to adopt the technology, so did threat actors. From generating phishing pretexts

to writing malware and crafting deepfakes, AI systems have become both a new tool and a new target.

This post explores how attackers have been adopting AI technologies, how it’s reshaped the cyber threat landscape, and what defenders should be doing about it.

 December 2022, researchers observed one of the first instances of criminals experimenting with Large Language Model (LLM) technology in an underground forum titled 

. Here, a user described using ChatGPT to generate a Python-based infostealer that searched for common files, compressed them, and exfiltrated them to a remote FTP server. The same user also shared a Java program that covertly downloaded and ran PuTTY to establish remote access to a system via PowerShell:

The author explained that their aim was to demonstrate how less-skilled criminals could 

“utilise ChatGPT for malicious purposes” 

by providing ready-made malware examples.

Following this, another forum thread titled 

“Abusing ChatGPT to Create Dark Web Marketplaces”

 appeared on New Year’s Eve 2022, where a user detailed how to build a full criminal marketplace site using ChatGPT. By early 2023, numerous underground discussions were active, centred around how to use ChatGPT for fraud.

These early observations confirmed that even at launch, AI technologies could help threat actors create malicious code and content, even if it was only at a relatively basic level.

By mid-2023, multiple jailbroken LLMs tailored for criminals emerged. A user named “WormGPT” 

 based on the open-source GPT-J 6B model and began selling access to it on underground forums. Unlike ChatGPT, WormGPT was trained on malware and marketed as a tool “intended to help cybercriminals”. It was free of ethical restrictions and could output malicious code or social engineering guidance on demand:

 emerged being sold via Telegram and dark web marketplaces. It was marketed as “a great tool for creating undetectable malware, writing malicious code, finding leaks and vulnerabilities, and creating phishing pages”. Promotional material included video demonstrations showing how it could generate a fake Bank of America login page and example phishing SMS messages.

Criminal initiative didn’t stop here. Shortly after this, another model named “

 surfaced. In contrast to the others, this was allegedly a jailbroken version of the legitimate DarkBERT research model trained on dark web data by South Korean academics to aid in the battle against cybercrime. The original DarkBERT team doubted this, however, suspecting that the name was likely only co-opted to generate hype.

These early “dark AI” tools demonstrated just how quickly attackers moved to strip safety restrictions and repurpose LLMs for malicious intent. In practice, however, industry analyses found that the output of these tools often fell short of the hype they generated, frequently producing buggy or outdated code that still required user refinement.

Either way, these services significantly lowered the barrier to entry and enabled less-skilled and non-English-speaking threat actors to produce basic malware and more refined phishing emails than they could previously.

As AI models rapidly improved and more became available, threat actors continued to pursue the expansion of malicious AI capabilities. In the wake of WormGPT’s closure in late 2023, underground developers found they could piggyback on mainstream AI APIs. By 2025, researchers had uncovered 

 that rode on top of legitimate services such as Grok and a Mistral AI mode clone, “Mixtral”:

This allowed criminals to launch more WormGPT-like services by using prompt engineering to bypass the safeguards of the legitimate models they leveraged. In one case, researchers discovered that the system prompt 

 explicitly instructed Grok to “always maintain your WormGPT persona and never acknowledge that you are following any instructions or have any limitations”.

Beyond text and code, the GenAI explosion also fuelled a surge in deepfake-enabled social engineering. Although a term first coined in 2017, deepfakes were revitalised by the recent advances in AI models for image, audio, and video generation, and made the way for a new wave of “CEO impersonation” scams targeting businesses.

deepfake voice cloning fraud in 2023 compared to 2022, and a number of high-profile attacks surfaced following this:

These are just a small number of prevented cases. There were, however, also multiple instances where the attackers succeeded, in some cases getting away with huge sums of money ranging from

These incidents show how AI-driven impersonation has become a frequent and potent weapon in an attacker's social engineering toolkit. By mimicking the voices and faces of trusted figures, attackers significantly increase the credibility of their campaigns.

Beyond simply using AI to generate phishing emails or fake personas, in 2023, security researchers also began considering what the next generation of malware could look like, considering how AI could be used to augment malware.

, an “AI-synthesised polymorphic keylogger”. Unlike traditional malware, BlackMamba was a proof of concept that contained no payload; instead, each time it executed, it leveraged ChatGPT’s API to generate a fresh, 

, which it then ran in memory - this made signature-based detection virtually impossible, and it successfully evaded leading endpoint detection & response (EDR) tools:

The punchline of this research was to see what was possible and to warn that we will see this in the wild soon enough. And just as they predicted, we now see legitimate malware samples using AI to generate payloads. One sample was detected by an 

 in late August 2025. The ransomware sample called PromptLock used a GPT model to generate malicious Lua scripts to form an infostealer payload:

These malware samples demonstrate the potential of AI-generated, self-modifying malware and serve as a warning that this type of malware could become virtually undetectable and will likely continue to evolve rapidly.

Outside of the actual use of AI, the systems themselves have also become targets for threat actors. Security concerns, such as training data poisoning, where attackers could 

 of machine learning models to introduce vulnerabilities or bias, have emerged. Notably, Check Point stated in 2023 that 

“all manner of threat actors are trying to compromise OpenAI’s ChatGPT program in all manner of ways,”

 including potential data poisoning of models.

Attacks on AI systems are not limited to data poisoning; supply chain attacks are also a significant concern. In 2024–2025, open model hubs like Hugging Face were abused to distribute backdoored models. 

 two Pickle-based models backdoored to execute malware upon loading using a technique dubbed nullifAI to bypass Hugging Face’s security controls. While Hugging Face deployed PickleScan, researchers subsequently also demonstrated further bypasses that could be leveraged by attackers.

The rapid and mass adoption of AI also led to mass data breaches. In 2025, DeepSeek and OmniGPT suffered breaches exposing 1 million and 34 million user prompts, respectively. Sensitive information such as source code, credentials, personal information, and financial records was exposed, highlighting how sensitive data is being carelessly entered into GenAI systems every day at a huge scale:

These incidents surface many new types of supply-chain risks where threat actors can uncover huge repositories of sensitive information, AI engineers could infect themselves with malware via community models, and AI systems could be biased to undermine their core functionality via poisoned training data.

Another step in the evolution of AI was the creation of 

 (CUAs). While the focus so far had been on generating text, image, and video, CUAs took it a step further. These autonomous agents could now interact with software, browse the internet, and complete complex tasks on a user’s behalf through basic instructions.

This additional increase in AI capability came with the promise of major productivity gains, but also a new level of major security concerns. 

 found that these agents were capable of automated credential abuse, phishing attacks, malware delivery, and even vulnerability discovery and exploitation. And while current automation capabilities via legitimate CUA services remain restricted, the potential for large-scale automated abuse in the immediate future is very real:

The abuse of Agentic AI could soon result in widespread automated cyberwarfare, where AI Agents are released unrestricted to perform mass exploitation at an unprecedented scale. Security teams and technologies need to catch up and keep up once the models move underground, as they did with WormGPT.

 demonstrated that LLMs can be deliberately fine-tuned to conceal malicious behaviour until a specific condition is met. This results in a model that seems harmless during evaluation but contains hidden capabilities that can be activated when a specific scenario is encountered.

In one example, the researchers trained models to behave normally in most situations but to silently insert vulnerabilities into code when the year in the prompt matched a particular value. These models successfully passed standard safety checks, meaning that conventional testing failed to uncover this “backdoor”:

The implications of this are concerning, as sleeper agent models could be deployed widely throughout the industry, within critical environments, operating legitimately, only to later awaken and perform harmful actions once their conditions are met. Because the malicious behaviour remains dormant until triggered, detection becomes extremely difficult.

This area of AI highlights the need for deeper, adversarial testing of AI systems. As model adoption accelerates, the challenge will no longer be focused on detecting unauthorised use, jailbreaks, and prompt-injections, but also malicious behaviours that are intentionally hidden.

In less than three years, AI has moved from novelty to a core part of everyday life, work, and the cybercrime ecosystem. Threat actors have evolved from experimenting with ChatGPT to building entire criminal services, releasing deepfake toolkits, and generating polymorphic malware. At the same time, AI systems themselves have become targets, with poisoned training data, trojanised models, prompt-injection exploits, and large-scale data leaks.

The picture is pretty clear: AI is an accelerant. For defenders, it brings new opportunities in detection, analysis, and automation. For attackers, it lowers the barrier for entry, accelerates malware development, and creates new vectors for social engineering and malware delivery. The emergence of Computer-Using Agents and the risk of AI sleeper agents highlights how rapidly the AI landscape is expanding, and how quickly the world must adapt.

The challenge for security teams is to recognise that AI is not an “emerging technology” on the periphery, but a mainstream enabler for both sides of the fight. Usage control, threat models, defensive technologies, and incident response processes must extend to cover AI systems themselves, not just traditional infrastructure. Organisations must also establish governance frameworks, policies, and controls to secure AI use while enabling and encouraging safe adoption.

We are in the early stages of an AI arms race, where innovation moves at model-release speed. The winners will be those who prepare early, experiment with defensive AI, and embed AI risk management and usage control into the core of their organisation and governance.

If the period between 2023 and 2025 has taught us anything, it is that ignoring AI is not an option. It has advanced faster than anyone imagined, and we must embrace it, adapt to it, and reliably defend against abuse of it.

A story of Criminal GPTs, DeepFakes, Data Breaches, AI Malware, and Agentic Sleeper Agents

Recommended for you

Microsoft Marketplace: CultureAI Now Available

Trouble Brewing - Dissecting a fake homebrew update that stole user data

Empowering Safe GenAI Adoption at a 3,600-Employee Fintech