Nine of Ten Steps: Why Agent Reliability Is an Evals Problem

Production SWAT — Status: Sev-1, ongoing. It's 2 AM. Production is down. Error rates are spiking across regions, transactions are failing, and the incident auto-escalates. A SWAT bridge spins up. Within minutes there are fifty people on the call — SREs, application teams, database on-calls, a duty manager reading status updates to leadership every fifteen minutes. Everyone is asking the same question: what changed?

While the bridge is still doing roll call, an AI agent has already picked up the incident.

The agent knows what to do. There's a runbook — the same one the SRE team has polished for two years. Ten steps, no shortcuts:

  1. Pull the incident record and confirm the affected service.
  2. Check the service's health endpoint.
  3. Check pod and container status in the cluster.
  4. Pull recent pod logs.
  5. Check recent deployments and config changes in the last hour.
  6. Check downstream dependencies — database, message queues, auth service.
  7. Check resource metrics: CPU, memory, disk pressure.
  8. Check ingress, DNS, and network reachability.
  9. Correlate against active platform-wide incidents.
  10. Write the root cause and remediation back to the ticket.

The agent has run this runbook two hundred times. It knows the shape of a memory leak, the fingerprint of a bad DB failover, the smell of an expiring certificate. Most nights it hands the on-call engineer a triaged ticket with the story already told and half the fix drafted.

Tonight, it does nine of the ten steps.

It skips step 5 — and writes back a confident, well-formatted root cause anyway: "Elevated latency on the primary database is causing upstream 500s in the affected service. Recommend failing over to the replica."

Had it checked the deploy history, it would have found that twenty minutes before the alert, someone shipped a config change that pointed the service at a stale environment variable. The database was fine. The failover would fix nothing — and bury the real cause deeper.

The agent's write-up lands in the bridge chat, and it sounds authoritative — so the bridge runs with it. The database team starts the failover. Fifty people spend the next twenty minutes chasing a database problem that doesn't exist. When someone finally scrolls through the deploy history, the fix is a one-line rollback. Under a minute, once anyone looks at the right thing.

The agent didn't lie. It skipped a step and didn't tell anyone.

The failure mode nobody greps for

Run that same agent on the next ten incidents and it nails all ten — every step, in order, grounded, correct. Then on the eleventh, for no reason you can point to, it drops step 5 again. Or step 7. Or cites a log line it never read.

This is not how traditional software fails. Skip a step in a bash script and you get an error, a non-zero exit code, something to grep for. Skip a step as an agent and you get a confident, plausible, ungrounded answer — one that reads exactly like the two hundred correct answers before it. Nothing looks broken. That's the failure mode.

When people say "AI is non-deterministic," this is what they mean. Not that the agent is random — that it's variable in ways you can't predict from the outside. The old QA reflex — did the output match what we expected? — falls apart, because the output almost always looks fine. The real question is different: how did the agent get here, and did it actually do the work?

The answer is one equation:

Reliability = Observability + Evals

Observability: the flight recorder

Observability means you can pull up any run and replay it — every tool call, every parameter, every response, in order. Drop tonight's incident into a trace and the gap is embarrassingly obvious:

get_incident → check_health → get_pods → get_pod_logs → (nothing) →
check_dependencies → get_metrics → check_ingress → check_platform_incidents →
update_incident

Between step 4 and step 6 there's a hole. get_recent_deployments was never called. Not "called and failed." Not "called and returned nothing." Never called. Without the trace, that hole is invisible — a resolved ticket looks the same whether the work was done or not.

What the trace shows
skipped — no error,
no log, no trace
1-4
done
5
never called
6-9
done
10
confident root cause
written anyway
What the runbook says
1-4
incident, health,
pods, logs
5
recent
deployments
6-9
deps, metrics,
ingress, platform
10
write root
cause

Evals: turning visibility into a verdict

Evals are QA for agents — scored the same way on every run, not just the ones someone happens to review. For an incident runbook, three matter most:

  • Answer relevancy — did the write-back address the incident, or answer a question nobody asked?
  • Groundedness — is the root cause backed by evidence the agent actually gathered in this run, or a plausible guess dressed up as a finding?
  • Tool sequence accuracy — did the agent follow the runbook, in order, without skipping load-bearing steps?

That third one catches tonight's miss. Step 5 was in the runbook; step 5 was not in the trace. The eval flags it before the engineer touches the ticket. Instead of "here's a confident root cause, trust me," the ticket reads: "9/10 steps completed — deployment history was not checked. Human verification recommended before acting."

No team can afford a senior engineer re-reading every ticket an agent closes, so a model does the reading. LLM-as-a-judge puts the agent's write-up side-by-side with the evidence it actually pulled and scores it: grounded or not, steps followed or not, answer on-topic or not. It's the same skim a senior engineer gives a ticket at handoff — except it runs on every ticket, not the two or three that get sampled.

10/10 steps,
evidence-backed
9/10 steps,
step 5 missing
Agent run
Observability trace
(every tool call,
every response)
Answer
relevancy
Groundedness
Tool sequence
accuracy
LLM-as-a-judge
Ticket closed —
15-second human verify
Flagged before handoff:
'deploy history not checked —
verify before acting'

The point is boring

Run the runbook on the next incident and the scores tell the story plainly: ten of ten steps present, root cause traced to a specific log line and a specific deploy hash, a ticket the engineer can verify in fifteen seconds. Nothing dramatic. That's the point.

Evals don't make an agent smarter. They don't stop it from having off nights. They make it impossible for a skipped step to pass as a complete one. The nine boring, correct runs stay boring. The one where the agent quietly dropped step 5 gets caught the night it happens — not next week, when someone reopens the ticket and realizes the failover fixed nothing.

An agent that can be caught is an agent you can trust with the pager.

Comments

Post a Comment

Popular posts from this blog

Evolution of Prompt Engineering to Context Engineering

Journey from Prompt to Context Engineering Generative AI has rapidly integrated into our daily work environment. Whether you're composing an email in Outlook, attending a meeting on Teams, drafting documents, or writing code in VS Code, intelligent copilots and assistants are now your companions. This revolution has fundamentally transformed how we interact with Large Language Models (LLMs), and the narrative around that interaction has evolved significantly. The Prompt Engineering Era When this AI revolution started, prompt engineering dominated the conversation. The prevailing wisdom was simple: the right prompt could accomplish anything. If the LLM wasn't responding as expected, the responsibility fell squarely on you—the prompt engineer—to refine your input. The approach was straightforward: improve your questions, and the LLM will provide better answers. We were encouraged to be more articulate, specific, and creative in how we framed our requests. As our confidence grew, ...
Read more
  The Hundred-Year Conversation: How We Spent a Century Learning to Speak Machine, and Then Machines Finally Learned to Speak Human July 2026 1926: The Ledger Imagine your great-grandfather at work. He sits at a wooden desk in a room full of other men at wooden desks. There are no computers. No screens. No phones. There is paper, ink, and the scratch of fountain pens. If he needs to calculate something, he does it in his head or on a mechanical adding machine — a heavy thing with rows of brass buttons that goes  cha-chunk  when you pull the lever. The work of the world is physical. Farmers drive horses. Factory workers pull levers. Clerks write in ledgers. The only "computer" in the building is a person — usually a woman with a sharp mind for arithmetic, employed in a typing pool to run numbers all day. The word "computer" literally means  a person who computes . Machines do physical work. They lift, carry, spin, and press. But they don't think. They don't c...
Read more

Personal Algo/AI Trading Engine

  What the Field Actually Does, and What to Build for Quick budge 1. The personal algo-trading landscape, 2015 → 2026 Retail algorithmic trading has lived through three distinct eras, and knowing which one you're joining matters more than picking a framework. 2015-2020 — the Quantopian era.  A single hosted platform, Quantopian, became the default: browser IDE, pre-loaded historical data, social leaderboard, paid contests where the firm allocated capital to winning strategies. At peak it had hundreds of thousands of users running millions of backtests on the same S&P 500 history [SRC-001]. That shared-dataset pattern was also what killed it. By pure statistics, thousands of "optimal" strategies emerged that were overfit to noise — they looked profitable in-sample and failed immediately in live trading. Quantopian shut down its live service in 2017, paper trading in 2019, contests in May 2020, and the whole community platform in November 2020. Robinhood quietly acquire...
Read more