If you have ever ridden in a self-driving car through a busy city, you may recognize the uneasy quiet that comes with it. There is no driver, no small talk, just a system making rapid judgments about the world outside. Most of the time, it works. Then the car brakes suddenly for a shadow or hesitates at an empty intersection, and the discomfort sets in. The problem is not the technology itself. It is the gap between confident action and human judgment. That same gap now defines one of the biggest risks facing enterprise AI.
Across industries, artificial intelligence is becoming more autonomous, faster, and more capable. Yet trust, not performance, is emerging as the deciding factor in whether AI succeeds or fails at scale. Many systems do what they are designed to do, but users are left unsure whether they should rely on the outcomes. When confidence falters, adoption follows.
Recent data illustrates how widespread the issue is. The MLQ State of AI in Business 2025 report found that 95 percent of early AI pilots fail to deliver measurable return on investment. The reason is rarely weak technology. More often, organisations deploy AI before they are clear on what problems they want to solve or who is responsible when something goes wrong. Leaders struggle to verify results, teams hesitate to trust dashboards, and customers lose patience when support feels automated rather than helpful. Anyone who has battled an automated customer service system that refuses to recognize a valid answer understands how quickly confidence can disappear.
Automation at scale, stability still uncertain
Klarna is often cited as a leading example of large-scale AI adoption. Since 2022, the company has cut its workforce roughly in half, saying its internal AI systems now perform the work of 853 full-time employees, up from 700 earlier this year. Over the same period, revenues have reportedly risen by 108 percent, and average employee compensation has increased by 60 percent, supported in part by efficiency gains from automation.
Kalyeena Makortoff
Yet the picture is not entirely positive. Klarna still posted a quarterly loss of 95 million dollars, and its chief executive has warned that more job cuts may follow. The case highlights a key point: automation can improve efficiency, but it does not automatically create resilience or trust. Without clear accountability and structure, operational gains can coexist with instability.
As Jason Roos, CEO of customer contact platform Cirrus, puts it, any transformation that undermines confidence inside or outside the business carries a cost that is easy to underestimate. Efficiency alone does not guarantee progress if people no longer trust the system delivering it.
When accountability is missing
History already offers examples of what happens when autonomy moves faster than oversight. In the UK, the Department for Work and Pensions used an algorithm to flag potentially fraudulent housing benefit claims. Around 200,000 cases were incorrectly identified, even though most were legitimate. The core failure was not the model’s logic, but the lack of clear ownership over its decisions. When automated systems suspend accounts or reject claims incorrectly, the most important question is not why the model failed, but who is accountable for the outcome.
Without a clear answer, trust becomes fragile. Users feel powerless, employees disengage, and organisations are left managing reputational damage rather than performance improvements.
Roos argues that readiness is the missing step in many AI strategies. If processes, data quality, and governance are not in place, autonomy does not fix problems. It magnifies them. The recommended sequence is straightforward: define the outcome, identify wasted effort, assess readiness and guardrails, and only then automate. Skipping those steps may deliver short-term gains, but accountability often disappears just as quickly.
Trust declines as scale increases
Part of the challenge lies in the race toward scale. Many organisations push for increasingly autonomous systems without fully considering how those systems behave when conditions change or edge cases appear. Public sentiment reflects that unease. The Edelman Trust Barometer shows a steady decline in trust in AI over the past five years. Separately, a joint study by KPMG and the University of Melbourne found that workers prefer greater human involvement in nearly half of the tasks examined.
The message is consistent. Trust does not come from pushing models harder or removing people entirely. It comes from understanding how decisions are made and ensuring governance guides systems rather than merely constraining them. Effective oversight acts more like a steering wheel than a brake.
The trust gap is equally visible on the customer side. PwC research shows a clear disconnect between how much trust executives believe they have earned and how customers actually feel. Transparency helps close that gap. Surveys consistently show that consumers want clear disclosure when AI is used in service interactions. When organisations are open about automation, customers are more likely to accept it as part of a blended human and digital experience. When they are not, people often feel misled.
Clarifying what “agentic AI” really means
Some confusion also stems from how agentic AI is described. It is often framed as unpredictable or self-directing, when in practice it is closer to structured workflow automation with reasoning and memory. These systems make limited decisions within boundaries set by people. The safest and most successful deployments follow a clear pattern: start with the outcome, map the workflow, assess readiness, and then select the technology. Reversing that order does not accelerate transformation. It simply produces mistakes at greater speed.
The broader lesson is that every major wave of automation eventually becomes a social issue, not just a technical one. Amazon’s success was built not only on logistics, but on the expectation that a package would arrive as promised. When that confidence weakens, loyalty fades. AI operates under the same logic. Sophisticated systems are meaningless if users feel confused, dismissed, or misled.
Internally, the stakes are similar. A global KPMG study shows how quickly employees disengage when they do not understand how automated decisions are made or who is responsible for them. Without clarity, adoption stalls, regardless of technical capability.
The human dimension of autonomy
As AI systems take on more conversational and customer-facing roles, emotional expectations matter more. Early feedback on autonomous chat experiences shows that people judge interactions not only on whether their issue was resolved, but on whether the exchange felt attentive and respectful. Frustration spreads quickly, and negative experiences rarely stay private.
The reality is that technology will continue to advance faster than people’s comfort with it. Trust will always lag innovation. That is not a reason to slow progress, but it is a reason to approach autonomy with care. Leaders should be able to explain an AI system’s last decision in plain language, define who steps in when it fails, and decide whether they would trust it with their own data.
As Roos succinctly puts it, agentic AI itself is not the threat. Unaccountable AI is.
When trust erodes, adoption follows, and ambitious projects become part of the failure statistics. Autonomy is not the enemy. Losing sight of responsibility is. Organisations that keep humans firmly involved in oversight are far more likely to remain in control long after the initial excitement around self-driving systems fades.
