Part – 2 of 10
Deepfakes & AI–Powered Impersonation
Artificial Intelligence (AI) is becoming embedded in the way Organizations operate, make decisions and serve customers. The fundamental question today is no longer “Can we use AI?” It is: “Can we use AI responsibly, securely and resiliently at scale?”
As AI becomes more capable and increasingly autonomous, it also becomes a new attack surface. What happens when an AI System / Solution is manipulated, impersonated, poisoned, stolen, compromised or simply goes wrong?
Financial Technology Frontiers (FTF)’s 10–part AI Security Risks Series examines the ten key AI Security Risks highlighted by the Canadian Centre for Cyber Security (CCCS), and goes beyond technology to explore the What, How, Impact and Control of each Risk. This series is designed to give Business Leaders and Technology, Cybersecurity, Risk and Compliance Professionals a practical framework for understanding and managing the many faces of AI Risk.
Deepfakes & AI–Powered Impersonation: When seeing and hearing is no longer believing
AI has fundamentally changed the economics of impersonation. A convincing voice, photograph or video can now be generated or manipulated at a fraction of the effort previously required. Welcome to Deepfakes!
What Is It All About?
Deepfakes use AI to create or manipulate audio, video or images to make an individual appear or sound as though they said or did something that never happened.
The CCCS identifies deepfake impersonation as a significant Business Risk and recommends strong identity verification techniques, phishing–resistant Multi–Factor Authentication (MFA), out–of–band verification and robust identity–binding processes for High–Risk functions. Examples include but are not limited to financial transactions, healthcare transactions, industrial controls and critical operations.
How Does This Threat Attack?
Attackers can collect publicly available photographs, videos and voice recordings of Executives, Employees or persons of key interests in Organizations, and use these artifacts to create convincing synthetic identities. The attacker can use these synthetic identities to convince victims through manipulation. Examples include:
- Making a fraudulent video call.
- Generating a fake voice instruction.
- Impersonating an Executive.
- Requesting a payment.
- Changing banking instructions.
- Obtaining sensitive information.
- Manipulating employees into bypassing normal procedures.
The real danger? Traditional “identity signals” such as voice, face and familiar communication style–can now be forged using AI.
What Harm Can This Risk Do?
Potential consequences include, but are not limited to:
- Fraudulent payments.
- Business Email Compromise (BEC).
- Theft of confidential information.
- Executive impersonation.
- Reputational damage.
- Customer fraud.
- Social engineering attacks.
- Loss of trust in corporate communications.
What Can Organizations Do To Control This Risk?
Organizations should consider moving beyond traditional identity and fraud controls and build AI–aware verification and authentication practices. Some of these include:
- Treating voice and video as untrusted identity signals. Organizations should consider introducing formal verification protocols for executive, customer and employee communications where AI–enabled impersonation could result in financial, operational or reputational harm.
- Adopting phishing–resistant MFA. Organizations should consider to progressively move critical users, privileged accounts and high–value transactions toward phishing–resistant authentication methods that do not rely solely on passwords, OTPs or easily spoofed identity signals.
- Establishing out–of–band verification for high–value transactions. Organizations should consider implementing independent verification channels for payments, changes to banking instructions, privileged–access requests and other high–risk activities.
- Introducing multi–person approval for sensitive financial activity. Organizations should consider strengthening segregation of duties and progressively increase approval requirements for transactions or instructions that could materially impact the Organization.
- Creating AI–aware verification procedures for unusual executive requests. Organizations should consider establishing predefined challenge–and–verify protocols for urgent, unusual or confidential requests, particularly those involving money, credentials, sensitive information or changes to established processes.
- Continuously training employees to recognize AI–enabled impersonation. Organizations should consider moving beyond conventional phishing awareness and conduct regular simulations involving deepfake video, synthetic voice, cloned executives and AI–generated social–engineering scenarios.
- Deploying capabilities to detect and investigate synthetic media. Organizations should consider to progressively incorporate deepfake detection, behavioural analytics, communication–pattern analysis and anomaly detection into fraud, SOC and identity–monitoring processes.
- Applying stronger adaptive identity controls to high–risk functions. Organizations should consider to using risk–based authentication, behavioural signals, transaction context and continuous verification for finance, privileged access, treasury, customer data and other sensitive operations.
FTF Analysis
The future control environment cannot rely on “seeing is believing” or “hearing is believing”. As AI makes identity increasingly easy to replicate, Organizations will need to shift from authenticating the person or communication to authenticating the intent, authorization and transaction. The mute question for Organizations is no longer simply, “Can we identify a deepfake?” It is, “Even if our identity signals are compromised, can our controls prevent an unauthorized action?”, Simply put, Organizations need to shift from asking “Is this really the CEO’s voice?” to “Is this transaction legitimately authorized?”. A convincing face or voice should never, by itself, authorize a payment or release sensitive information. Authentication must increasingly shift to the transaction level.
About This Article:
This Article explores the growing risk of Deepfakes and AI–Powered Impersonation as AI makes it increasingly possible to replicate voices, faces, identities and executive communications with remarkable realism. It examines how attackers can exploit these capabilities to deceive employees, bypass traditional identity–verification practices, manipulate transactions or gain access to sensitive information. The Article highlights the potential Cybersecurity, Fraud, Operational and Business Risks arising from AI–enabled impersonation and examines the forward–looking controls Organizations can adopt to strengthen identity, authentication and transaction–level verification in an increasingly synthetic world.
This Article is authored by, Narasimham Nittala who leads the Strategy and Research vertical of Financial Technology Frontiers (FTF). This Article is published as part of FTF’s Hi2AI Series. Like our previously published Articles, this Article is written in an accessible, practitioner–focused format and it aims to raise awareness about responsible AI adoption across various Institutions.
FTF believes that service providers, Fintech entities, consulting firms and technology companies can all benefit from reflecting on the perspectives shared here and consider how their own approaches to AI Risk Management can evolve. Practitioners in various industries are equally encouraged to adapt these insights to their unique contexts.
About Hi2AI
Hi2AI is FTF’s AI ecosystem For Financial Services. Hi2AI is a trusted community shaping the future of Artificial Intelligence in Financial Services by driving responsible innovation, influencing policy with regulators, and crafting future standards that ensure growth, resilience, and trust across the global financial ecosystem. Hi2AI exists to accelerate the responsible adoption of AI across the global financial ecosystem through:
- AI–Driven Industry Collaboration.
- Ecosystem Connection & Innovation.
About Financial Technology Frontiers
Financial Technology Frontiers (FTF) is a global media–led fintech platform dedicated to building and nurturing innovation ecosystems. We bring together thought leaders, financial institutions, fintech disruptors, and technology pioneers to drive meaningful change in the financial services industry.
References
- Canadian Centre for Cyber Security – Top 10 AI Security Actions
- NIST – Generative AI Risk Management Profile
- NIST AI Risk Management Framework
