When AI Gets an Identity, Who Takes Responsibility?

Lessons from the Hi2AI Security Edition Roundtable on Identity, AI Agents, Accountability and Trust

Artificial intelligence is moving rapidly from answering questions to accessing Systems, making decisions and acting on behalf of people. That evolution is forcing Financial Services to reconsider one of the most fundamental concepts in Cybersecurity: Who or what is “Identity” now? And more importantly, How do Organizations establish Identity, Trust and Accountability when machines begin acting on behalf of humans?

This was the central question explored at Financial Technology Frontiers (FTF)’s Hi2AI Security Edition in Toronto on 22 September 2026, during the Roundtable “Can We Still Trust Identity in an AI World?”.

Participants at the Hi2AI Security Edition Roundtable in Toronto, 22 September 2026

This Article captures the essence of the discussions and presents the thoughts of the participants of the Roundtable.

Identity Is No Longer Just About People

Traditionally, Identity has meant a person, Account or credential. The Roundtable identified a much broader Identity landscape emerging with AI: AI Agents, APIs, applications, machine identities and autonomous Systems–all potentially capable of accessing resources and taking action.

Narasimham Nittala (NN), Strategy & Research Leader at FTF moderated the Roundtable by initiating the frame of the conversations: “Identity must increasingly be understood as something that can belong to both humans and non–human entities”.

This seemingly simple shift has profound implications for Financial Services. If a machine can access Information, initiate a transaction or decide, Organizations must know what that entity is, what authority it has, whom it represents, and who is Accountable for its actions.

1. From Human Identity to Machine Identity

The Roundtable identified three increasingly important forms of Identity:

  • An Agent operating through its own Identity or service Account.
  • An AI Agent acting with authority delegated by a human.
  • The traditional human Identity.

The challenge is that Organizations must now govern all three simultaneously. This brings least privilege, zero Trust and Identity governance into a new environment–particularly as technologies such as MCP and plugins enable Agents to interact with Enterprise Systems. But the Roundtable identified a more fundamental problem: How do you secure an Agent if you don’t even know it exists?

2. The Shadow–AI Problem

The visibility challenge is becoming an extension of the familiar Shadow IT problem. Organizations may know the applications and Systems they have formally approved, while having limited visibility into shadow AI, shadow Agents and autonomous tools introduced by employees. The Roundtable identified a new requirement: Organizations need to know:

  • Who created the AI Agent?
  • Who owns the AI Agent?
  • What can the AI Agent access?
  • On whose behalf can the AI Agent act?
  • What data can the AI Agent see?
  • What decisions can the AI Agent make?
  • What happens when the AI Agent’s permissions (or the Agent itself) changes?

This suggests that AI Identity And Access Management (IAM) cannot simply be another IAM capability. It needs to become part of a broader AI Accountability Architecture.

3. AI Is Changing the Fraud Equation

The Identity challenge extends beyond Enterprise access. The Roundtable also explored how AI is transforming Impersonation, Phishing, Synthetic Identity and Brand Fraud. AI can make fraudulent communications increasingly convincing while allowing attackers to scale Impersonation. This is pushing Organizations towards proactive protection of their broader Digital Identity, including Domains, Social Accounts, Websites and other Digital Assets.

The objective is shifting from detecting Fraud after the customer interacts with it to identifying Impersonation and Fraudulent Digital Identities before they reach the customer.

4. The Real Question: Accountability

Perhaps the strongest Theme emerging from the Roundtable was that Identity ultimately comes down to Accountability. The discussion reframed Identity from a question of mere Authentication to something much larger: Who is answerable for an action?

This becomes particularly important when an AI Agent acts on behalf of a human. Imagine an AI Agent selecting an Insurance Policy, executing a Payment, recommending an investment or negotiating a transaction. If that AI Agent is developed, hosted and configured by different parties, where does responsibility sit?

  • The customer?
  • The Organization?
  • The developer?
  • The infrastructure provider?
  • The Organization that configured the Agent?

The Roundtable identified delegated authority as a critical emerging concept. Giving an AI Agent autonomy should not mean that the AI Agent gets unrestricted authority. Permissions, boundaries, transaction limits, escalation mechanisms and human oversight will increasingly become essential components of AI security, especially in an Agentic AI environment.

5. AI Challenges the Traditional Responsibility Model

The Roundtable then confronted an even harder problem: Non–Deterministic Behaviour. Traditional Technology environments are generally built around predictable System behaviour and defined shared–responsibility models. AI is different. The same instruction can potentially produce different outcomes.

As the discussion highlighted, an Organization may define what it wants an AI System to do, yet the System may behave differently across multiple scenarios. That creates a new Accountability question: If the Organization provides the instruction, but the AI produces an unexpected result, who is responsible?

The answer may ultimately extend beyond technology and Cybersecurity into contracts, regulation, governance and the courts.

6. Trust May Become the Real AI Architecture

The discussion ultimately moved beyond Identity management. The deeper issue is the Architecture of digital Trust. As autonomous AI becomes embedded into Financial Services, four elements must remain connected:

Identity → Authority → Accountability → Auditability

The question is no longer simply: Is this Identity genuine? It transcends to:

  • Who does this Identity represent?
  • What authority has been delegated to that Identity?
  • What can the Identity do?
  • And who is Accountable for what the Identity does?

7. Wrap–Up

The Roundtable’s frame, “Identity as an evolving concept that one must encompass humans, machines and autonomous entities” helps move the conversation beyond traditional IAM towards the larger Architecture of Trust and Accountability in an AI–driven Financial Ecosystem.

 A speaker addressing the Hi2AI Security Edition at ICICI Bank Canada, Toronto

8. The Next Frontier

The Hi2AI Security Edition was designed around emerging questions of AI Attack Surfaces, Machine Identities, AI–enabled Fraud, Governance, Third–Party Risk and Digital Trust. The Identity Roundtable demonstrated that these issues cannot be treated independently.

  • AI security is increasingly becoming a question of Identity.
  • Identity is becoming a question of authority.
  • Authority is becoming a question of Trust.
  • And Trust ultimately depends on Accountability.
  • The technology may be autonomous.
  • Responsibility cannot be.

Acknowledgements

Financial Technology Frontiers (FTF) acknowledges the insightful contributions of the participants of this discussion, without which these outcomes could not have been inked.

FTF thanks its Partners for enabling the Hi2AI Security Edition, and making it a grand success:

Related Posts