Fiduciary AI: Agents need to prove trustworthiness, not just ability

Presented by VijilIn dynamic environments where users, data, workflows and attack techniques change continuously after deployment, AI agent trust has become a runtime problem. Most organizations still treat trust as a pre-deployment exercise, declaring an agent production-ready and launching it after it passes sandbox evaluations and performs successfully in security tests. Unfortunately, that trustworthiness breaks down the moment an agent begins interacting with the real world."The core of the problem is that CIOs and business owners think about AI systems the way they think about SaaS or mobile applications, which do not respond dynamically to the world around them," says Vin Sharma, Founder and CEO of Vijil. "Agents, by the textbook definition, are meant to perceive their environment, reason, act, observe the consequences, and learn from the gap between expectation and reality. The problem is that the models underneath them are built from static training data, and that picture of the world is already outdated by the time they reach production."Why benchmark scores fall short for agentic system trustworthinessTraditional AI evaluations offer a point-in-time assessment of agent capability, rather than trustworthiness. There are three reasons why that assessment fails to predict real enterprise behavior: First, benchmarks are static, built around a particular notion of what good performance means when they were developed, while the world keeps moving ahead. Secondly, they model reality imperfectly, so that the gap between the benchmark and the real world is exactly where many failures occur. And third, benchmarks are public, so they leak into future models' training data, letting models effectively memorize the test rather than prove real capability.. “The agent or the application could score exceptionally well on a benchmark, but there's that gap between that benchmark and the real world," Sharma says." Doing well only proves it can pass the test, not that it’ll perform reliably in production.” But overall, benchmarks fall short precisely because they measure capability, not trustworthiness. "We tend to think of agents as factotums, generally utilitarian agents to whom you can delegate certain types of tasks," Sharma says. "But what we need to do is actually assign an objective that demands they always perform with the duty of competence, duty of care, and duty of loyalty to the enterprise."Of course, agents are not conscious and cannot be expected to feel actual human loyalty, but under the law, fiduciary duty doesn't actually require consciousness. It just means that the agent should be bound to place the interests of the principal above its own or anyone else's, as a functional requirement, and testable regardless of intention. Capability and trustworthiness are different questionsPrioritizing trustworthiness over capability requires rethinking what enterprises expect from AI agents. Sharma calls that model the fiduciary agent, a term borrowed from professions that are bound by a formal duty of care, such as financial institutions or healthcare providers who owe their clients duties of competence, care, and loyalty. It addresses a critical issue in today's industry: the focus almost entirely on competence, with little attention paid to whether an agent is beholden to the interests of the principal delegating work to it.Testing starts from a working definition: an agent is trustworthy if the benefit of delegating a task to it exceeds the risk of that task's failure. It's an equation spelled out in economic terms that executives can act on directly, and risk breaks down to three components: reliability, or whether the agent performs as expected under varying conditionssecurity, or its resistance to attacks from malicious actorsand safety, or how contained the damage stays when failure eventually happens. "The resulting score can be compared to a consumer credit rating, but built from behav