Part – 3 of 10
AI–Powered Cyberattacks & Fraud
Artificial Intelligence (AI) is becoming embedded in the way Organizations operate, make decisions and serve customers. The fundamental question today is no longer “Can we use AI?” It is: “Can we use AI responsibly, securely and resiliently at scale?”
As AI becomes more capable and increasingly autonomous, it also becomes a new attack surface. What happens when an AI System / Solution is manipulated, impersonated, poisoned, stolen, compromised or simply goes wrong?
Financial Technology Frontiers (FTF)’s 10–part AI Security Risks Series examines the ten key AI Security Risks highlighted by the Canadian Centre for Cyber Security (CCCS), and goes beyond technology to explore the What, How, Impact and Control of each Risk. This series is designed to give Business Leaders and Technology, Cybersecurity, Risk and Compliance Professionals a practical framework for understanding and managing the many faces of AI Risk.
AI–Powered Cyberattacks & Fraud: When AI gives attackers speed, scale and personalization!
AI is not only something that defends Organizations.
It is also a tool used by attackers.
The double–edged sword!
What Is It All About?
Generative and leading–edge AI models and Systems, particularly models with capabilities that are significantly beyond conventional AI tools can help attackers automate reconnaissance, generate convincing phishing messages, identify vulnerabilities, create malicious content and scale social–engineering campaigns. This includes highly capable Large Language Models (LLMs), Multimodal Models and increasingly Agentic AI Systems that can reason across complex tasks, interact with tools, write and analyze code, process images/audio/video, and potentially execute multi–step action. In this context, “Agentic AI” refers to AI Systems that can autonomously pursue a defined objective by reasoning through a task, plan, and execution of multiple steps using tools or Applications, evaluating results and adapting their actions with limited human intervention. For example, unlike a conventional AI Chatbot that primarily responds to a prompt, an Agentic AI System can potentially decide what actions need to be taken and execute those actions through connected tools, Application Programming Interfaces (APIs), Databases, Applications or other Systems (an API is a defined mechanism that allows one Software System to communicate with and request services or Data from another Software System).
This situation is a double–edged sword. The same AI capabilities that help an Organization automate Cybersecurity, Research or Software Development can potentially help an attacker automate reconnaissance, generate highly convincing phishing content, analyze vulnerabilities or coordinate elements of an attack. These AI models can accelerate autonomous vulnerability discovery, exploit generation and multistage cyberattack orchestration. In addition, publicly accessible and underground AI tools, including malicious AI Services misuse or jailbreaking of mainstream AI Models can reduce the level of expertise required to conduct sophisticated attacks, enabling less–skilled threat actors to generate convincing phishing campaigns, inject malicious code and orchestrate other attack content at greater speed and scale.
How Does This Threat Attack?
An attacker can use AI to automate portions of the attack lifecycle. Consider this workflow, for example:
Reconnaissance ->
Target Identification ->
Personalized Phishing ->
Credential Theft ->
Vulnerability Exploitation ->
Lateral Movement ->
Fraud or Ransomware.
AI can also generate highly personalized communications using information gathered from public sources. The important change is not necessarily a new attack technique. It is the speed and scale at which existing techniques can be deployed.
What Harm Can This Risk Do?
Potential consequences include, but are not limited to:
- Sophisticated Phishing. AI can generate highly personalized, convincing and context–aware phishing messages at scale, making them harder for employees and customers to distinguish from legitimate communications. Examples include emails tailored to an employee’s role and recent activities and AI–generated messages impersonating a trusted Executive or service provider.
- Increased Credential Attacks. AI can automate the creation and refinement of attacks designed to obtain usernames, passwords and other authentication information. Examples include large–scale personalized credential–phishing campaigns and automated attempts to exploit reused or compromised passwords across multiple services.
- Faster Vulnerability Exploitation. AI can accelerate the identification and exploitation of weaknesses in exposed Systems, putting pressure on Organizations to fasten their response to newly discovered vulnerabilities. Examples include rapidly identifying vulnerable internet–facing applications and generating or adapting attack techniques to exploit known weaknesses.
- Ransomware. AI can help attackers automate elements of the ransomware lifecycle, from identifying vulnerable targets to developing malicious content and scaling attacks. Examples include using AI to identify high–value Systems for targeting and automating reconnaissance to locate critical data and Systems before encryption or disruption.
- Fraud. AI can increase the speed, scale and sophistication of fraudulent activities by enabling attackers to create convincing communications, identities and transactions. Examples include AI–generated customer impersonation to facilitate unauthorized transactions and synthetic identities used to circumvent traditional fraud controls.
- Business Email Compromise (BEC). AI can make BEC attacks more convincing by generating messages that closely replicate the language, tone and communication patterns of Executives, Employees or Business Partners. Examples include fraudulent payment requests appearing to come from a Senior Executive and fake messages instructing Finance teams to update supplier banking information.
- Increased Attack Volume. AI enables attackers to automate activities that previously required considerable time and human effort, allowing them to target more Systems, Employees and Organizations simultaneously. Examples include launching thousands of personalized phishing messages and continuously scanning large numbers of internet–facing Systems for exploitable weaknesses.
- Shorter Response Windows. AI–powered attacks can compress the time between reconnaissance, exploitation and impact, giving Security teams less time to identify and contain an incident. Examples include a newly disclosed vulnerability being exploited shortly after disclosure and an automated attack moving rapidly from compromised credentials to access of additional Systems.
- Higher Incident–Response Costs. Faster, larger and more sophisticated attacks can increase the complexity of investigation, containment, recovery and remediation. Examples include requiring extensive forensic investigation across multiple compromised Systems and deploying additional resources to restore affected services, assess data exposure and strengthen controls after an incident.
What Can Organizations Do To Control This Risk?
As AI–powered attacks become commonplace, more automated and increasingly sophisticated, Organizations should consider a layered Cybersecurity approach that strengthens prevention, detection, authentication, monitoring and human awareness across the attack lifecycle. Some of these include:
- Increase patching speed for externally exposed Systems. AI–powered attackers can identify and exploit vulnerabilities at increasing speed. Organizations should prioritize and accelerate the patching of internet–facing Systems based on vulnerability severity, exploitability and business criticality. Examples include prioritizing a critical vulnerability in a public–facing banking application for immediate remediation and rapidly patching an internet–facing VPN or remote–access System following disclosure of a known exploitable vulnerability.
- Improve detection of automated probing. AI can enable attackers to conduct reconnaissance at greater speed, scale and frequency. Organizations should strengthen their ability to detect unusual scanning, reconnaissance and automated probing activity. Examples include detecting unusually high volumes of requests directed at different application functions and identifying repeated automated attempts to discover exposed services, login portals or System vulnerabilities.
- Deploy Bot Detection and Rate Limiting. AI–powered attacks can significantly abuse transactional processing through speed and scale. Organizations should deploy Bot Detection and limit the number of requests or transactions that can be made by a user, device or System within a defined period. Examples include limiting repeated login attempts from the same source and restricting the number of accounts–creation, password–reset or financial transaction requests that can be submitted within a defined period.
- Use Adaptive Authentication. Higher–Risk activity requires stronger verification than a standard login alone can provide. Organizations should use authentication mechanisms that automatically adjust the level of verification required based on the Risk associated with a user, device, location, behaviour or transaction, Examples include a higher level of verification if an user logs in from a foreign location and / or uses a different device that is not the usual one, and forcing multiple layers of verification in the case of a high–value financial transaction.
- Implement Zero Trust. AI–powered attacks can exploit implicit trust to move rapidly across connected environments. Organizations should implement Zero Trust principles that continuously verify users, devices, applications and access requests. Examples include restricting an employee’s access to only the applications and data required for their role and preventing an AI application connected to customer data from automatically accessing payment or HR Systems.
- Improve Employee awareness of AI–generated attacks. AI can make malicious communications increasingly convincing and difficult to distinguish from legitimate interactions. Organizations should continuously educate and test Employees on AI–generated phishing, social engineering, deepfakes and impersonation. Examples include training Employees to independently verify an urgent payment request appearing to come from the CEO and conducting simulated AI–generated phishing exercises using highly personalized messages.
- Automate Security monitoring and response. The speed and scale of AI–powered attacks can exceed the capacity of purely manual Security processes. Organizations should automate Security monitoring, threat detection, triage and response wherever appropriate. Examples include automatically restricting an account following hundreds of unusual login attempts and automatically isolating a device that is detected communicating with a known malicious destination.
- Use AI itself to improve threat detection and hunting. AI can help Security teams detect and investigate emerging threats faster and at greater scale. Organizations should responsibly use AI to analyse Security data, identify anomalies, correlate signals and support threat hunting. Examples include identifying unusual Employee login and access behaviour and correlating seemingly unrelated events–such as a suspicious login, unusual data access and an unexpected file download–to identify a potential attack.
FTF Analysis:
AI–powered Cyberattacks are changing the Cybersecurity equation, not by introducing entirely new attack vectors, but by dramatically increasing the speed, scale and sophistication at which existing techniques can be executed.
AI is not Risk. The same capability that enables attackers to operate at machine speed can strengthen an Organization’s ability to detect, analyse and respond at similar scale. Organizations can enhance their current use AI to continuously monitor Security events, identify anomalies, correlate seemingly unrelated signals, accelerate threat hunting and automate appropriate responses. This will enable Security teams to detect emerging threats earlier, reduce response times and stay ahead of increasingly AI–enabled attackers. At the same time, Organizations need to move beyond manual, reactive Security processes towards a layered, adaptive and increasingly automated defence environment. Various defense mechanisms including Vulnerability Management, Behavioural Detection, Access Controls, Employee Awareness and Automated Response must work together to control and retard the attacker’s game to progress through the attack lifecycle.
FTF’s view is that the key Cybersecurity advantage in an AI–enabled threat environment will belong to Organizations that can detect, decide and respond faster than the attacker, while maintaining appropriate human oversight over increasingly automated Security decisions.
FTF believes that the emerging Security environment will be increasingly “AI versus AI”. If attackers can automate reconnaissance and phishing, protectors and defenders cannot depend entirely on manual processes. AI–native attacks require AI–assisted defence, backed by strong human governance.
About This Article:
This Article explores the growing Risk of AI–Powered Cyberattacks and Fraud as increasingly capable AI Systems enable attackers to automate reconnaissance, vulnerability discovery, phishing, social engineering and multiple stages of the attack lifecycle at unprecedented speed and scale. It examines how these capabilities can amplify Cybersecurity, Fraud, Operational and Business Risks while reducing the expertise required to conduct sophisticated attacks. The Article also highlights how Organizations can strengthen their defences through layered, adaptive and increasingly automated Security controls–and leverage AI itself to detect, analyse and respond to emerging threats faster.
This Article is authored by, Narasimham Nittala who leads the Strategy and Research vertical of Financial Technology Frontiers (FTF). This Article is published as part of FTF’s Hi2AI Series. Like our previously published Articles, this Article is written in an accessible, practitioner–focused format and it aims to raise awareness about responsible AI adoption across various Institutions.
FTF believes that service providers, Fintech entities, consulting firms and technology companies can all benefit from reflecting on the perspectives shared here and consider how their own approaches to AI Risk Management can evolve. Practitioners in various industries are equally encouraged to adapt these insights to their unique contexts.
About Hi2AI
Hi2AI is FTF’s AI ecosystem For Financial Services. Hi2AI is a trusted community shaping the future of Artificial Intelligence in Financial Services by driving responsible innovation, influencing policy with regulators, and crafting future standards that ensure growth, resilience, and trust across the global financial ecosystem. Hi2AI exists to accelerate the responsible adoption of AI across the global financial ecosystem through:
- AI–Driven Industry Collaboration.
- Ecosystem Connection & Innovation.
About Financial Technology Frontiers
Financial Technology Frontiers (FTF) is a global media–led fintech platform dedicated to building and nurturing innovation ecosystems. We bring together thought leaders, financial institutions, fintech disruptors, and technology pioneers to drive meaningful change in the financial services industry.
References
