Artificial Intelligence (AI) is becoming embedded in the way organizations operate, make decisions and serve customers. The fundamental question today is no longer “Can we use AI?” It is: “Can we use AI responsibly, securely and resiliently at scale?”
As AI becomes more capable and increasingly autonomous, it also becomes a new attack surface. What happens when an AI System / Solution is manipulated, impersonated, poisoned, stolen, compromised or simply goes wrong?
FTF’s 10–part AI Security Risks series examines the ten key AI Security Risks highlighted by the Canadian Centre for Cyber Security (CCCS), and goes beyond technology to explore the What, How, Impact and Control of each Risk. This series is designed to give Business Leaders and Technology, Cybersecurity, Risk and Compliance Professionals a practical framework for understanding and managing the many faces of AI Risk.
Part – 1 of 10
Prompt Injection Risks
When an AI System or Solution can be manipulated into doing what it was never supposed to do
AI Systems are designed to follow instructions. That is precisely what makes Prompt Injection dangerous. An attacker can manipulate those instructions and cause the AI System to behave outside its intended purpose.
What Is It All About?
Prompt Injection occurs when malicious instructions are inserted into an AI interaction to influence the System’s behaviour. A jailbreak is a related technique intended to bypass the System’s safety restrictions or policy controls.
The Risk becomes substantially greater when an AI System is connected to enterprise data, applications, APIs, software tools or autonomous agents. The CCCS recommends input sanitization, protection of system prompts, output filtering, identity controls, restrictions on high–risk tools and validation of downstream actions.
How Does This Threat Attack?
An attacker may place malicious instructions directly into a prompt, or indirectly into information that an AI System retrieves. For example, an employee asks an AI Assistant to summarize a document. The document contains hidden instructions telling the AI to ignore its original task and retrieve confidential information. This is particularly dangerous in Retrieval–Augmented Generation (RAG) and Agentic systems because the malicious instruction can originate from external content rather than the user.
NIST specifically distinguishes direct and indirect Prompt Injection and notes that indirect attacks can potentially lead to proprietary–data theft or remote code execution.
What Harm Can This Risk Do?
Potential consequences include:
- Confidential–data leakage.
- Unauthorized access.
- Malicious code execution.
- Manipulation of AI–generated decisions.
- Unauthorized transactions.
- Compromise of connected applications.
- Reputational damage. and
- Regulatory or privacy exposure.
What Can Organizations Do To Control This Risk?
Organizations can consider several control activities including but not limited to:
- Treat all external content as untrusted.
- Separate instructions from retrieved data.
- Limit AI access to sensitive information.
- Apply least–privilege permissions to AI agents.
- Validate AI–generated commands before execution.
- Filter and quarantine anomalous outputs.
- Restrict external communications by AI Systems.
- Red–team AI applications regularly.
FTF Analysis
The biggest mistake is treating Prompt Injection as merely a “prompt problem.” The real issue is what the AI System is allowed to do after it has been manipulated. A chatbot with no privileges is a relatively contained Risk. But an AI agent capable of moving money, changing records or executing code represents a very different Risk category.
About This Article:
This Article explores the growing risk of Prompt Injection as organizations increasingly embed AI into applications, workflows and decision-making processes. It examines how malicious or unintended instructions can manipulate AI systems into bypassing safeguards, revealing sensitive information or taking actions beyond their intended purpose. The Article highlights the potential Cybersecurity, Operational and Business Risks arising from such attacks and examines the controls organizations can adopt to securely deploy and govern AI Systems.
This Article is authored by, Narasimham Nittala who leads the Strategy and Research vertical of Financial Technology Frontiers (FTF) and is published as part of FTF’s Hi2AI Series. Like our previously published Articles, this Article is written in an accessible, practitioner–focused format and it aims to raise awareness about responsible AI adoption across various Institutions.
FTF believes that service providers, Fintech entities, consulting firms and technology companies can all benefit from reflecting on the perspectives shared here and consider how their own approaches to AI Risk Management can evolve. Practitioners in various industries are equally encouraged to adapt these insights to their unique contexts.
About Hi2AI
Hi2AI is FTF’s AI ecosystem For Financial Services. Hi2AI is a trusted community shaping the future of Artificial Intelligence in Financial Services by driving responsible innovation, influencing policy with regulators, and crafting future standards that ensure growth, resilience, and trust across the global financial ecosystem. Hi2AI exists to accelerate the responsible adoption of AI across the global financial ecosystem through:
- AI–Driven Industry Collaboration.
- Ecosystem Connection & Innovation.
About Financial Technology Frontiers
Financial Technology Frontiers (FTF) is a global media–led fintech platform dedicated to building and nurturing innovation ecosystems. We bring together thought leaders, financial institutions, fintech disruptors, and technology pioneers to drive meaningful change in the financial services industry.
References
- Canadian Centre for Cyber Security – Top 10 AI Security Actions
- NIST – Generative AI Risk Management Profile
- OWASP – Prompt Injection
