AI–Powered Cyberattacks & Fraud
Artificial Intelligence (AI) is becoming embedded in the way Organizations operate, make decisions and serve customers. The fundamental question today is no longer “Can we use AI?” It is: “Can we use AI responsibly, securely and resiliently at scale?”
As AI becomes more capable and increasingly autonomous, it also becomes a new attack surface. What happens when an AI System / Solution is manipulated, impersonated, poisoned, stolen, compromised or simply goes wrong?
Financial Technology Frontiers (FTF)’s 10–part AI Security Risks Series examines the ten key AI Security Risks highlighted by the Canadian Centre for Cyber Security (CCCS), and goes beyond technology to explore the What, How, Impact and Control of each Risk. This series is designed to give Business Leaders and Technology, Cybersecurity, Risk and Compliance Professionals a practical framework for understanding and managing the many faces of AI Risk.
AI Red Teaming & Security Testing
You cannot secure an AI System you have never tried to break
AI Systems behave differently from conventional software. Traditional Penetration Testing alone may therefore miss important AI–specific vulnerabilities.
What Is It All About?
Security Testing and AI Red Teaming are related but distinct activities. According to the National Institute of Standards and Technology (NIST), Security Testing verifies whether an implementation protects Data and maintains functionality as intended. AI Red Teaming goes a step further by deliberately challenging an AI System through structured, often adversarial testing to uncover flaws, vulnerabilities, unexpected behaviours and potential misuse. In practice, Security Testing establishes whether protections work as intended, while AI Red Teaming actively explores how those protections might be circumvented. Both are essential to identifying weaknesses before attackers exploit them and to strengthening the security and resilience of AI Systems.
AI Red Teaming involves evaluating AI Systems, Pipelines and Interfaces against attacks such as Prompt Injection, Evasion and Poisoning, and updating controls and guardrails as attacks evolve.
How Does This Threat Attack?
Attackers continuously search for ways around AI safeguards. They may attempt to:
- Bypass Content Controls. Attackers may craft inputs that circumvent an AI System’s safety rules, filters or restrictions. This can cause the AI System to generate content that it was designed or configured not to generate.
- Manipulate System Behaviour. Attackers may provide carefully crafted instructions or inputs that alter how an AI System responds or makes decisions. The objective is to make the AI System behave differently from its intended design.
- Poison Data. Attackers may deliberately introduce inaccurate, misleading or malicious information into Data used to train, fine–tune or operate an AI System. This can influence the AI System’s future outputs or decisions.
- Extract Sensitive Information. Attackers may manipulate an AI System into revealing confidential information contained in its Prompts, Training Data, Connected Sources or Conversations. This could include Personal, Financial, Proprietary or other Sensitive Information.
- Cause Unsafe / Unreliable Outputs. Attackers may manipulate an AI System into producing inaccurate, harmful or inappropriate recommendations or instructions. In a connected environment, such outputs could potentially lead to unsafe decisions or actions.
- Exploit Connected Tools. An AI System may be connected to Applications, Databases, APIs or other Enterprise Tools. Attackers may attempt to manipulate the AI into misusing these connections or performing actions beyond its intended authority.
- Induce an AI Agent to perform Unintended Actions. AI Agents can plan and execute multiple steps using connected tools and Applications. Attackers may manipulate an AI Agent into taking actions that were not intended or authorized by the Organization.
- Manipulate RAG Sources. Attackers may compromise or introduce misleading information into the external knowledge sources used by Retrieval–Augmented Generation (RAG) approach. The AI System may then retrieve and rely on this manipulated information when generating its responses. RAG is an AI approach where an AI System retrieves relevant information from external sources–such as documents, databases or knowledge repositories–and uses that information to generate its response.
A Security Test conducted only once at deployment would become obsolete as Systems, Data, Dependencies and Attack Techniques change. Hence the need for continuous testing and monitoring.
What Harm Can This Risk Do?
Inadequate or infrequent testing can leave AI–related weaknesses undiscovered until attackers exploit them. Potential consequences include, but are not limited to:
- Undiscovered Vulnerabilities. Weaknesses in AI Models, Applications, Pipelines or connected Systems may remain hidden until they are discovered and exploited by attackers. This can leave Organizations exposed without adequate time to prepare or respond.
- Data Leakage. An inadequately tested AI System may unintentionally disclose confidential, personal or proprietary information. Such leakage can occur through Prompts, Outputs, Connected Data Sources or Compromised Integrations.
- Unsafe Autonomous Actions. An AI Agent with insufficient Controls may take actions that were not intended or properly authorized. In Connected Environments, this could result in inappropriate transactions, changes to AI Systems or other operational consequences.
- Regulatory Exposure. AI failures involving Privacy, Security, Discrimination, Financial Activity or other regulated processes may create Compliance and Regulatory concerns. Organizations may also face difficulty demonstrating that appropriate Controls and Testing were in place.
- Compromised AI Applications. Unidentified weaknesses can allow attackers to manipulate an AI Application, its Data, its Outputs or its connected capabilities. The AI System may then become a pathway for broader Cybersecurity compromise.
- Operational Disruption. Manipulation of an AI System or its supporting Infrastructure can interfere with business processes and critical workflows. This may reduce availability, interrupt Operations or require significant remediation.
- Loss of Customer Trust. AI–generated errors, inappropriate actions or Security Incidents can undermine confidence in an Organization’s use of AI. Repeated or high–impact incidents can damage both customer relationships and the Organization’s reputation.
What Can Organizations Do To Control This Risk?
AI Security cannot be established simply by securing the AI System at the point of deployment. Organizations need to recognize that AI Systems, their Data, Models, Interfaces and Connected Capabilities will be challenged by increasingly sophisticated attackers and therefore need to be continuously tested under realistic adversarial conditions. Red Teaming should become an ongoing validation process that identifies weaknesses before they are exploited, tests whether existing guardrails work and ensures that controls evolve as the AI System, its environment and the threat landscape change. Organizations should:
- Include AI–Specific Scenarios in Red–Team Exercises. Organizations should design Red Team exercises that specifically test AI–related attack scenarios, rather than relying only on conventional Cybersecurity testing. These scenarios should reflect how attackers could manipulate AI Models, Prompts, Data, Agents and connected Systems.
- Test Direct and Indirect Prompt Injection. Organizations should test whether AI Systems can be manipulated through both direct user inputs and indirect instructions embedded in external content. Testing should assess whether malicious instructions can override safeguards or influence the System’s behaviour.
- Test Data–Poisoning Scenarios. Organizations should test whether manipulated, inaccurate or malicious Data can influence AI Models or their outputs. This should include assessing the integrity of Data Sources, Training Data, RAG repositories and other information used by the AI System.
- Test Connected Tools and Agents. Organizations should test whether AI Agents can be manipulated into misusing Connected Applications, APIs, Databases or other Enterprise Tools. Testing should verify that Agents operate only within their authorized permissions and intended boundaries.
- Continuously Update Security Guardrails. Organizations should regularly review and strengthen AI Security guardrails as new vulnerabilities, attack techniques and AI capabilities emerge. Controls that were effective at deployment may become inadequate as the AI System and threat environment evolve.
- Conduct Adversarial Testing before Major Releases. Organizations should perform structured adversarial testing before deploying significant changes to an AI Model, Application or Agent. This helps identify Security weaknesses before new functionality is exposed to Employees, Customers or external users.
- Repeat Testing after Material System or Data Changes. Organizations should re–test AI Systems whenever there are significant changes to Models, Data, Prompts, Applications, APIs, Agents or underlying infrastructure. Testing should also be triggered by material changes in the AI System’s intended use or operating environment.
- Incorporate AI Incidents into Tabletop Exercises. Organizations should include AI–specific scenarios in Cybersecurity and Incident–Response Tabletop Exercises. This allows Teams to practise how they would respond to events such as Prompt Injection, Data leakage, compromised AI Applications or an AI Agent taking unintended actions.
FTF Analysis
AI Security Testing should move from a point–in–time penetration test to a continuous adversarial validation. The AI System is not the only object being tested. The entire AI Ecosystem–Data, Models, Systems, Prompts, APIs, Agents, Users and Downstream Systems–must be tested as an integrated attack surface.
Organizations must not only verify that their AI Security Controls work but actively challenge those Controls to discover how they might fail. The objective of AI Red Teaming is therefore not simply to demonstrate that an AI System is secure under expected conditions, but to deliberately explore how an attacker could manipulate its Data, bypass its Guardrails, influence its outputs, exploit its Connected Capabilities or induce unintended actions.
This represents an important shift in the traditional Cybersecurity mindset. For conventional IT Systems, Security Testing may establish whether defined Controls function as intended. For AI Systems, Organizations increasingly need to ask a more challenging question: “What happens when an AI System encounters inputs, Data, instructions or circumstances that its designers did not anticipate?”
AI Systems are dynamic. Models evolve, Data changes, Applications are modified, new APIs and Tools are connected, AI Agents acquire new capabilities, and Attack Techniques continue to adapt. A System that passes a Security Test today may present a completely different risk tomorrow. Security Assurance must consequently become an ongoing cycle of “Test, Identify, Remediate, Validate and Retest”.
FTF believes that AI Red Teaming should ultimately become part of an AI System’s Lifecycle–not an activity performed only before deployment or after an Incident. Major Model changes, new Data sources, new Tools or APIs, changes in AI System behaviour and significant changes in intended use should all trigger appropriate Adversarial Testing.
The real measure of AI Security is therefore not whether an AI System has passed a Test, but whether the Organization has the capability to continuously discover how an AI System can fail, understand the resulting risk and strengthen its Controls before those weaknesses are exploited.
AI is not only something that defends Organizations. It is also a tool used by attackers. Essentially, it is a double–edged sword. Organizations that gain the greatest advantage will be those that use AI not only to build capabilities, but also to continuously challenge, test and strengthen those capabilities.
About This Article:
This Article explores the importance of AI Security Testing and Red Teaming as Organizations increasingly deploy AI Systems across critical business processes and applications. It examines how Security Testing can validate whether AI Systems and their Controls operate as intended, while Red Teaming deliberately challenges those AI Systems to uncover vulnerabilities, unintended behaviours and potential avenues of misuse before attackers exploit them. The Article also highlights how Organizations can move from point–in–time Security Assessments to Continuous Adversarial Validation, strengthening the resilience of AI Systems as Models, Data, Applications, Agents and Attack Techniques evolve.
This Article is authored by, Narasimham Nittala who leads the Strategy and Research vertical of Financial Technology Frontiers (FTF). This Article is published as part of FTF’s Hi2AI Series. Like our previously published Articles, this Article is written in an accessible, practitioner–focused format and it aims to raise awareness about responsible AI adoption across various Institutions.
FTF believes that service providers, Fintech entities, consulting firms and technology companies can all benefit from reflecting on the perspectives shared here and consider how their own approaches to AI Risk Management can evolve. Practitioners in various industries are equally encouraged to adapt these insights to their unique contexts.
About Hi2AI
Hi2AI is FTF’s AI ecosystem For Financial Services. Hi2AI is a trusted community shaping the future of Artificial Intelligence in Financial Services by driving responsible innovation, influencing policy with regulators, and crafting future standards that ensure growth, resilience, and trust across the global financial ecosystem. Hi2AI exists to accelerate the responsible adoption of AI across the global financial ecosystem through:
- AI–Driven Industry Collaboration.
- Ecosystem Connection & Innovation.
About Financial Technology Frontiers
Financial Technology Frontiers (FTF) is a global media–led fintech platform dedicated to building and nurturing innovation ecosystems. We bring together thought leaders, financial institutions, fintech disruptors, and technology pioneers to drive meaningful change in the financial services industry.
References
