Why Enterprise AI Agents Fail Silently, and How to Make Them Trustworthy

In May 2026, a team running a production support agent published a number that should worry every enterprise AI owner. For nine days their agent was wrong about roughly one ticket in fourteen, and not a single dashboard noticed. The workflow completed. The status codes were clean. The replies read well. By every signal the monitoring stack could emit, the system was healthy. It was also lying to them.

This is not a story about one badly built agent. It is the signature failure pattern of the whole category. Enterprise AI agents almost never fail the way software fails, loudly, with stack traces and pager alerts. They fail the way a confident but mistaken employee fails, by producing work that looks right, sounds right, and is wrong. And because the entire point of an agent is to take a human out of the loop, the person best placed to catch the mistake is exactly the person you removed.

What follows is a look at why that happens, grounded in the failure research of the last two years, and a practical blueprint for the engineering and governance layers that turn agents from plausible liabilities into systems an enterprise can actually trust.

Part 1 · The state of the field

The numbers behind the agent boom

Big spend, high cancellation rates, and a diagnosis that points at engineering, not models.

Two findings frame the moment. Gartner projects that more than 40% of agentic AI projects will be cancelled by the end of 2027, pointing to rising costs, unclear business value, and thin risk controls. Of the thousands of vendors claiming agentic capability, Gartner reckons only around 130 offer the real thing, and has a name for the rest: agent washing. MIT's NANDA initiative, in its much-discussed GenAI Divide report, found that despite $30 to 40 billion in enterprise GenAI spending, roughly 95% of pilots produced no measurable P&L impact. Only a small set of deeply integrated systems created real value.

The interesting part is what these studies do not blame. Gartner's analysts are clear that the failures come from hype-led deployment and missing risk controls, not raw model capability. MIT's authors point to a learning gap, systems that cannot retain feedback, adapt to context, or improve, rather than to model quality. Both land on the same diagnosis. The intelligence is arriving faster than the engineering discipline around it.

Meanwhile the same analysts expect the category to keep growing. Gartner forecasts that by 2028 agentic AI will sit inside a third of enterprise software and make 15% of day-to-day work decisions on its own. The agents are coming either way. The open question is whether they arrive inside systems built to notice when they are wrong.

Part 2 · Anatomy of a silent failure

Agents don't crash. They confabulate.

Why a healthy dashboard and a broken agent can be the very same agent.

Reliability engineering already has a name for the most dangerous class of production failure: gray failure, where a component degrades while its failure detectors keep reporting health. The underlying property is differential observability. The application suffers, but the observer meant to catch it does not.

LLM agents inherit that whole problem and then make it worse. A 2026 longitudinal study of a production agent runtime, 22 incidents with full root-cause postmortems, backed by more than 4,000 unit tests and 800 governance checks, found one pattern showing up at least 28 times: a failure whose error signal never reaches a human in a form they can act on. The authors sorted the causes into five mechanism classes: environment quirks, design-assumption mismatches, error swallowing and dilution, chained hallucination and fabrication, and forensic blind spots. The fourth is unique to LLM systems, and the most dangerous. When an upstream error leaks into an agent's context window, the failure mode is not silence. It is plausible speech. The model turns the error into a fluent narrative and hands it to the user with full confidence. The authors gave it a name that deserves a place in every platform team's vocabulary: fail-plausible.

Their numbers should change how teams think about agent QA. Around 70% of silent failures were caught by a human reading the output. Unit tests caught almost none of this class. Incident latency ran from 13 hours to 60 days, and it tracked the failure mechanism rather than code complexity. The longest-lived failures hid in the seams between components, where no single owner was watching. A retrospective audit prevented none of the incidents before the fact, but blocked 87% of regressions after. Audits, in other words, are regression engines, not prediction engines.

Why do conventional stacks miss all of this? Because everything we have built for the last fifteen years watches for mechanical failure: 500s, timeouts, out-of-memory kills, queue depth. An agent that quietly loses context, misreads one tool argument at step 2, or drifts off its goal throws none of those signals. It returns a 200 and the wrong answer, which is exactly the blind spot. Tool misuse is now widely reported as the most common agent-specific failure in production, and its signature is nasty: one malformed argument early in a run silently corrupts every downstream step that depends on it. Your dashboard is not lying to you. It is faithfully answering a question that stopped being the right one the moment you put an LLM in the loop. Not "did it respond?" but "was it right?"

Part 3 · The math nobody runs

Per-step brilliance, end-to-end coin flips

The arithmetic that turns impressive per-step accuracy into unreliable workflows.

The second reason agents fail quietly is not linguistic. It is arithmetic. An agent workflow is a chain of dependent steps: plan, retrieve, call a tool, parse, reason, write, act. If each step succeeds with probability p, an n-step workflow succeeds with probability p to the power of n. Multiplied, not averaged. The result is brutal and deeply counterintuitive.

A step reliability of 95%, a number most teams would happily celebrate, gives you a 36% success rate across a 20-step workflow. Even 99% per step, which is genuinely hard to hit against real enterprise data, leaves you at 82% end-to-end at 20 steps, and it keeps decaying from there. This is the exact mechanism behind the great-demo, broken-rollout pattern. Demos are short, happy-path chains over clean data. Production is long chains over messy reality. Researchers at METR made the time version of this concrete, showing that agent success decays roughly exponentially with task duration, so each model has something like a task half-life. The Illusion of Diminishing Returns paper formalised the flip side: past a high threshold, small gains in single-step accuracy buy hyperbolic gains in the task length you can actually complete. Reliability at the step level is the highest-leverage investment in the whole stack.

One good run doesn't mean it works

There is a subtler place the same math hides, in how success gets measured. Most published benchmark scores answer one question: if we give the agent a few tries at a task, did it get it right at least once? Researchers call this pass@k. That is a fair way to measure what an agent is capable of, but it is the wrong question for a business. If your agent handles a customer ticket perfectly on Monday and botches the identical ticket on Wednesday, "it worked at least once" is no comfort. The question that matters is stricter. If we hand the agent the same task ten times, does it get it right all ten? Researchers at Sierra built a benchmark, tau-bench, around exactly that. They call the stricter score pass^k. Read it as passed every time.

The gap between the two is enormous. When tau-bench tested leading agents on realistic customer-service tasks, they solved fewer than half on the first attempt. Asked to solve the same tasks eight times in a row without a single miss, success fell below one in four. The rule of thumb is simple. An agent that gets it right 90% of the time on a single run gets through eight repeats flawlessly only about 43% of the time. So an agent your team demos successfully once is not an agent your enterprise can hand a process to.

Part 4 · Where failures actually come from

It's the system, not the model

Most failures start in how the system was specified, not in how smart the model is.

When agents do fail, where does it start? The best evidence comes from MAST, the Multi-Agent System Failure Taxonomy, built by Berkeley researchers from more than 1,600 annotated execution traces across seven popular agent frameworks, with expert annotators reaching strong agreement (kappa = 0.88). MAST identifies 14 distinct failure modes, and the way they cluster is the headline. Roughly 42% of failures trace to specification and system design: ambiguous task definitions, poorly drawn agent roles, missing termination conditions. Another 37% come from inter-agent misalignment: agents ignoring each other's inputs, losing context at handoffs, proceeding on mismatched assumptions. Only about 21% land in task verification, the category people instinctively blame, and even there the failure is usually a verifier that was absent or superficial, not a model that could not be checked.

Sit with what that distribution means. Nearly four in five agent failures are things a platform team controls: how tasks are specified, how roles are bounded, how handoffs carry context, when workflows are forced to stop. This has already been validated the constructive way. Follow-up work that imposed explicit behavioural contracts, structured messaging, and lifecycle verification on multi-agent pipelines cut total failures by as much as 70% on some software-engineering tasks, with no model upgrade at all. The instinct to wait for the next frontier model to fix reliability is exactly backwards. Better models raise the per-step ceiling. Only architecture stops errors from compounding and slipping out silently.

Part 5 · The adversarial dimension

Silent failure is a security surface too

The same silence that hides bugs also hides attackers.

Everything so far assumes good faith, an agent failing on its own. But the same properties that make agents fail silently make them exploitable silently, and in December 2025 OWASP formalised this with its Top 10 for Agentic Applications, a taxonomy built by more than a hundred security researchers. The list reads like a catalogue of quiet disasters: goal hijacking through direct and indirect prompt injection (ASI01), tool misuse via unsafe delegation (ASI02), identity and privilege abuse across delegation chains (ASI03), memory and context poisoning that reshapes behaviour long after the first interaction (ASI06), cascading failures that widen the blast radius across connected agents (ASI08), and the one most relevant here, human-agent trust exploitation (ASI09), where confident, polished output leads a human operator to approve something harmful.

Notice what they share. None of them announce themselves. A poisoned memory entry looks like a memory entry. A hijacked goal is pursued with the same fluent diligence as a legitimate one. Microsoft's AI Red Team reported that memory poisoning needed only a single successful injection to persist across sessions. And ASI09 is the fail-plausible mechanism turned into a weapon. The agent's greatest strength, sounding certain and competent, becomes the attacker's delivery vehicle. Any serious answer to how we make agents trustworthy has to treat reliability and security as one problem with one set of controls, because to the enterprise they look identical: work that seemed right and wasn't.

Part 6 · The fix

The Trust Stack: five layers between an agent and an incident

Five concrete layers that make an agent's mistakes visible, bounded, and recoverable.

Trustworthy agents are not a model property. They are a system property, produced by layered controls that assume the agent will sometimes be wrong and make wrongness visible, bounded, and recoverable. Across the research and the production postmortems, the controls settle into five layers.

Layer 1. Specification as contract

Since the biggest share of failures happens before the first token is generated, treat agent specifications the way you treat API contracts. Every agent gets an explicit goal, an explicit boundary ("you do not touch X"), machine-checkable success criteria, and the item the MAST data shows teams neglect most, a termination condition. An agent that cannot say when it is done is an agent that will decide for itself, and its decision will occasionally be an infinite loop billed by the token. Protocol-driven work built on MAST showed that contracts plus structured inter-agent messaging, on their own, remove a majority of failures on some task families.

Layer 2. Least privilege, budgeted autonomy

Scope every tool to the minimum the task needs. Read-only where possible, per-resource where feasible, separate credentials per agent so delegation chains cannot quietly escalate. Sandbox code execution. Cap budgets, steps, tokens, wall-clock, spend, so the worst case of a runaway loop is a bounded invoice rather than an unbounded incident. This is the OWASP mitigation layer, and it doubles as a reliability layer. A blast radius you have constrained is a failure you can afford to detect slowly.

Layer 3. Independent verification at the seams

The compounding math tells you where the checks go: at every handoff, because that is where errors both enter and multiply. Cheap deterministic validators first, schema checks, referential checks against the source system, invariant assertions such as "the reconciled total must equal the ledger total." Then semantic verification, a separate verifier agent or consensus across independent attempts, for steps whose correctness cannot be reduced to a rule. The critical property is independence. A verifier that shares the worker's context inherits the worker's blind spots. And verify the verifiers. The silent-failure study's darkly funniest incident was a governance audit that printed "all invariants hold" over a pile of checks that had themselves crashed. The observer had no observer.

Layer 4. Observability built for semantic failure

This is the layer that specifically kills silent failure, and it has two halves. The first is tracing: every run captured as a tree of spans, prompt construction, retrievals, model calls, tool invocations with their arguments and results, policy checks, final output, using the emerging OpenTelemetry GenAI semantic conventions (gen_ai.*) so the data is vendor-neutral and lines up with the rest of your APM. When a bad output surfaces at step 9, the trace is what lets you walk back to the malformed argument at step 2 in minutes instead of days. This is not abstract. In Eerly AI Studio, an agent's actions across connected systems, an SAP write, a Workday lookup, a generated document, each land as a traced span tied to the person who triggered it, so a wrong result at step nine walks back to its origin without a forensic dig. The second half is evaluation. Traces tell you what happened. Evals tell you whether it was any good. Run automated judges on a meaningful sample of production traffic, score outputs against task-specific rubrics, cluster failures into issues rather than drowning in per-event logs, and keep humans grading a slice of what the judges grade, because judge models drift too. Remember the base rate. In the production study, around 70% of silent failures were caught by human eyes on the output. Your evaluation layer's job is to industrialise that scrutiny, not to replace it.

Layer 5. Human oversight where it counts

Human-in-the-loop everywhere is an expensive interface for a human decision-maker. Human-in-the-loop nowhere is how you end up in a postmortem. The answer is risk-tiered autonomy. Classify actions by reversibility and blast radius. Let the agent run free on low-stakes reversible work, require approval for anything irreversible or externally visible, and make escalation a first-class agent capability. An agent that says "I'm not confident, here's what I have" should be rewarded by the workflow, not punished by it. Design the approval surface honestly too. OWASP's ASI09 exists because polished agent output rubber-stamps itself through tired reviewers. Show the human the evidence and the trace, not just the fluent summary.

Part 7 · Closing

Trust is the product

Trust is an engineering outcome, not a model you buy.

The pattern across every study here is the same. Agents don't fail because the models are dumb. They fail because the systems around them were built to assume success. The dashboard watched for crashes and got fluency. The benchmark measured potential when the business needed consistency. The pilot measured a demo when production was a distribution.

That reframing is, in the end, optimistic. Specification failures, coordination failures, verification gaps, missing telemetry, unbounded privileges: these are engineering problems, and engineering problems yield to discipline. The enterprises crossing the divide in the MIT data are not the ones with secret model access. They are the ones treating agents as production software inside real workflows, with contracts, permissions, verification, observability, and honest human oversight wrapped around a loop they assume, cheerfully and permanently, to be fallible.

At Eerly, this is the thesis the platform is built on. Agents that act across enterprise systems within the permissions teams already have, with policy-level control over what agents can reach, and execution that is traceable end to end. Silent failure is the default state of an unengineered agent. Trust is what you get when you refuse to accept the default.

Sources & further reading

• Cemri, M. et al.: Why Do Multi-Agent LLM Systems Fail? (MAST taxonomy, UC Berkeley / NeurIPS 2025) — arxiv.org/abs/2503.13657
• When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime (arXiv, 2026) — arxiv.org/abs/2606.14589
• Yao, S. et al.: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Sierra, 2024, introduces pass^k) — arxiv.org/pdf/2406.12045
• Gartner: Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 (2025)
• MIT NANDA: The GenAI Divide: State of AI in Business 2025
• OWASP GenAI Security Project: Top 10 for Agentic Applications 2026, and the AI Agent Security Cheat Sheet
• The Illusion of Diminishing Returns: Measuring Long-Horizon Execution in LLMs — arxiv.org/pdf/2509.09677
• Towards Engineering Multi-Agent LLMs: A Protocol-Driven Approach (SEMAP) — arxiv.org/pdf/2510.12120
• Latitude: Detecting AI Agent Failure Modes in Production; Towards AI: Silent AI Agent Failures Are the Production Risk No Dashboard Catches
• OpenTelemetry: GenAI Observability with OpenTelemetry (GenAI semantic conventions)