In brief
- AI adoption over 2024 and 2025 outpaced governance, with malicious open-source packages up 156% year on year and most security teams unable to say where LLMs are running.
- Attackers now use AI for polymorphic malware, supply chain poisoning, and faster social engineering, cutting average breakout times to around 18 minutes.
- Regulation is hardening: the EU AI Act, ISO/IEC 42001 and NIST's draft AI profile all push organisations towards demonstrable control.
- Practical first steps include building an AI inventory, controlling shadow AI usage, securing the model supply chain, and updating detection and response.
Introduction
2026, the perfect time to reflect on how far technology has come and what lies ahead. Without a doubt, Artificial intelligence has gone from a niche to an omnipresent force, reshaping how we work, build, and defend. While organisations have speed-ran the adoption of AI and machine learning, cybercriminals have been just as fast to exploit them, and AI now powers business decisions, customer interactions, and, predictably, cyberattacks.
In this post, I’ll reflect on what changed over the past two years, and look ahead to how we can prepare, secure our environments, and make sense of the AI-driven world we now operate in.
2024–2025: A Rapidly Changing World
The last two years saw an explosion in AI adoption. Businesses integrated AI models into their day-to-day operations, users and employees integrated it into their daily lives, and threat actors leveraged AI to improve their attacks. And the attackers have quite a lot to show for it:
A Surge in AI Supply Chain Attacks
Software supply chain breaches escalated with the rise of AI. The number of malicious open-source packages has increased 156% year-over-year, with a growing focus on the poisoning of AI libraries and models. JFrog’s 2025 report documented over 1 million new models added to Hugging Face in 2024 alone, with a 6.5x increase in malicious models uploaded by attackers.
AI-Generated Malware and Evasive Threats
Threat actors harnessed AI to create polymorphic and context-aware malware, able to rewrite itself and adapt to the target environment. Security researchers noted that modern malware can now automatically evade sandbox environments and blend into normal operations semantically, complicating detection.
Faster and Smarter Social Engineering
Attackers also used AI to improve their social engineering efforts. For example, AI-driven voice cloning made phone scams far more convincing, with voice phishing attacks having increased 442% by late 2024, according to CrowdStrike data. Auto-generated spear-phishing emails, deepfake audio and video, and choreographed AI scam campaigns increased in frequency and by mid-2025, criminals were breaching networks and starting to move laterally in just 18 minutes on average, down from ~1h previously.
Uncontrolled AI Adoption and Shadow Systems
As organisations rushed to deploy AI (outpacing governance and security controls), a 2025 survey found that 62% of security teams could not identify where LLMs were running, and 68% of organisations allowed employees to deploy AI technologies without approvals. This lack of oversight led to widespread “shadow AI”, exposing sensitive data and introducing unvetted models and technologies into critical systems. The impact of this was evident in an EY report showing that half of organisations have already suffered AI-related security issues, with only 14% of CEOs believing that AI adequately protects sensitive data.
Early Response Regulation and AI-Aware Security
By late 2025, we’d started to see a few adjustments to this new reality. Forward-leaning organisations adopted AI-aware security tooling, including implementing scanners for malicious models, and AI security and governance platforms (like CultureAI) to manage secure adoption across the workforce. These tools provided visibility into shadow AI use, enabled sanitisation of prompts, and supported policy enforcement around AI use and data handling.
Meanwhile, industry and regulatory bodies accelerated their efforts to formalise AI risk management:
- Safe deployment practices: Security teams began favouring formats like SafeTensor to avoid arbitrary code execution during model loading, moving away from vulnerable formats such as Pickle.
- The EU Artificial Intelligence Act: Established risk-based compliance requirements for AI systems. This requires high-risk AI systems to meet strict documentation, risk management, and post-market monitoring obligations or face fines reaching up to €35 million or 7% of global revenue.
- ISO/IEC 42001: The world’s first AI management system standard offered a structured governance framework to support responsible AI development, deployment, and oversight.
- NIST AI Cybersecurity Profile (Draft): An extension of the NIST Cybersecurity Framework (CSF 2.0) emerged that provides guidance for managing AI-related cybersecurity risk across three focus areas: securing AI systems, enhancing cyber defence with AI, and preparing for AI-enabled threats.
By the end of 2025, AI had clearly redefined the threat landscape, and the organisations best positioned to adapt were those that treated AI governance, usage control, and secure-by-default tooling as essential, not optional overhead.
2026: An AI-Dominated Future
We now understand the scale of the challenge, and 2026 will be about operationalising that understanding, putting frameworks, tooling, and governance into practice before the next wave of attacks arrives. For security leaders, preparation is no longer about distant future proofing. It is about acting now to protect against AI-driven threats that already exist.
So, based on the lessons of ~2024 -2025, what are a few clear priorities that have emerged for 2026 and beyond?
AI Attacks Will Escalate
Attackers are increasingly using AI for polymorphic malware, model poisoning, and automated attacks. Defenders should expect continuously adapting threats and shift away from static signatures towards behaviour-based detection. A large-scale AI model supply chain breach is plausible in 2026. Organisations with model inventories and integrity checks will be best placed to respond quickly.
Regulation and Governance Will Converge
AI regulation will become an enforceable reality. The EU AI Act is in force, and the Cyber Resilience Act will require SBOMs for software, including AI components. In parallel, organisations will increasingly align their internal AI governance with emerging standards such as ISO/IEC 42001. As a result, AI risk management will become a standard part of enterprise risk frameworks, driven by regulators, customers, and insurers alike.
Defenders Will Use AI With Caution
AI-powered security tools will see wider adoption to cope with the scale and speed of AI-driven threats and adoption. This will include malware analysis, model scanning, and AI security and governance solutions. This will help organisations enable AI use while preventing data leaks, enforcing policy, and providing guardrails around everyday AI interactions. Human oversight will remain critical. As defensive AI can be manipulated or poisoned, organisations must adopt a human-in-the-loop model with skilled analysts in control, continuously validating AI outputs.
Frameworks and Community Will Mature
Beyond regulation and internal governance, the broader AI security ecosystem will continue to mature. NIST is expected to finalise its AI Cybersecurity Profile, while industry groups expand shared tooling for model transparency, AI asset/dependency inventory generation, and vulnerability scanning. These developments will lower the barrier to adoption and create greater consistency across organisations. AI-specific requirements are likely to be embedded into security policies, assurance processes, and training as part of normal security operations.
Securing AI-Driven Environments
Taken together, these trends point to a clear direction of travel. AI-driven threats will intensify, regulation will harden, and expectations around governance and control will rise regardless of organisational readiness. The question for security leaders is no longer whether these changes will matter, but how quickly they can be operationalised.
For organisations that have not yet put these capabilities in place, the following steps outline practical ways to begin securing AI-driven environments.
- Create an AI Inventory: Catalogue all AI models, services, and integrations, including third-party and unofficial use. Even a basic inventory covering ownership, purpose, data inputs, and review status significantly improves visibility and risk awareness.
- Control Shadow AI and AI Usage: Establish clear policies for approved AI tools and acceptable use, supported by AI security and governance solutions that enforce policy at the point of interaction. These controls help prevent sensitive data disclosure through user prompts, apply guardrails to everyday AI use, and provide visibility into how AI tools are being used across the organisation. Training should reinforce safe usage patterns and awareness of AI-enabled risks such as data leakage and deepfakes.
- Secure the AI Supply Chain: Treat external models and AI components as third-party code. Verify integrity, cache trusted models internally, and restrict internet access from production environments. Some model formats can execute code and should be treated as potentially hostile.
- Use Safer Model Formats: Avoid executable formats such as Pickle where possible and standardise on non-executable alternatives like SafeTensors. Where legacy formats remain, document risk, apply compensating controls, and plan migration.
- Embed Security Across the AI Lifecycle: Integrate threat modelling, adversarial testing, access controls, and usage monitoring into AI development and deployment. Test for prompt injection, data leakage, and abuse cases. Restrict who can deploy and query production models, and ensure all interactions are logged and auditable.
- Update Detection and Response: Adapt monitoring and incident response to AI-specific threats, including anomalous model behaviour and AI-enabled data exfiltration. Train SOC teams on AI-related attacker techniques and rehearse AI supply chain and model compromise scenarios.
Taken together, these steps apply established security principles to AI systems. They reduce risk while enabling confident adoption of AI. Organisations that build trust and resilience into their AI environments will be better positioned both defensively and competitively in the years ahead
Conclusion
AI has become embedded in the very fabric of modern society, shaping how work is done, how people think, and how attacks unfold. The rapid adoption of AI over the past few years revealed huge gains, but often without the visibility, governance, and controls required to manage the newly introduced risks. Attackers adapted quickly, AI supply chains emerged as viable targets, and many organisations found their users operating AI systems that were unaccounted for and unvetted.
The challenge now is no longer about anticipating AI’s impact but enabling the secure operation of an AI-dominated environment. Frameworks are maturing, and expectations from regulators focus on the need for demonstrable control and accountability.
Organisations that are succeeding in 2026 are those enabling AI safely rather than slowing adoption, investing in visibility, usage control, and secure-by-default practices across the AI lifecycle. Human oversight remains essential, as effective defence still depends on informed judgement alongside automation.
AI is no longer an emerging technology to be planned for. It is a present reality that must be governed, secured, and operated with intent.