Agent evaluation harness: grade the run, not the reply
Pull-quote: “A correct answer produced by a broken run is a defect you have not found yet.”
An agent evaluation harness grades runs, not replies, because an agent is a process and the reply is only its last step. A single model call produces one artifact you can score. An agent produces a trajectory, and its final answer may not reflect anything that happened along the way.
This post is for engineers who have an agent running and no dependable way to tell whether a change made it better. You will end with a metric set, a task suite, fixtures that make runs repeatable, and a merge gate that fails on a named task rather than a drifting average. It assumes the vocabulary of earlier posts: the perceive-reason-act-verify loop, the context budget, and typed tool contracts.
What you will be able to do
- Grade a run against a verifiable end state instead of scoring the final message.
- Instrument the trajectory metrics that locate where a run degraded.
- Build a task suite with difficulty tiers, regression tests, and tasks that should be refused.
- Make runs repeatable with recorded tool responses, sandboxed writes, and frozen memory.
- Use a model judge only where it is legitimate, with its known biases controlled.
- Gate merges on named regressions and report calibration honestly.
Why does grading the final answer mislead you about agents?
Grading the final answer misleads you because an agent’s answer and an agent’s process fail independently.
A false pass: the agent picked the wrong tool twice, hit a rate limit, recovered nothing, then wrote a fluent summary from what it recalled of a similar task. Nothing was verified, and against a different record that run fabricates.
A false fail is the reverse. The API returned 503, the record genuinely did not exist, or the task needed a permission the agent properly declined to escalate. A response grader marks all three as failures, and optimising against that signal teaches an agent to guess instead of to refuse. Two runs emitting identical text can carry opposite value: one verified a database write, the other asserted it.
A trajectory is the ordered record of every model input, tool call, tool result, verification check, and state change produced by one attempt at one task. Grade that record and response quality becomes one metric among several.
Which trajectory metrics show where a run went wrong?
Nine metrics cover most of the diagnostic need, and only the first concerns the answer.
| Metric | What it measures | A bad reading means |
|---|---|---|
| Task completion | The end state holds afterwards | The outcome failed, with no clue where |
| Steps to completion | Loop iterations per task | More work, same result |
| Tool-call validity rate | Calls that parse against the schema | A schema regression |
| Tool-selection accuracy | Calls matching the expected tool | Naming collisions between tools |
| Verification failures | Actions the verify stage rejected | An unreliable act stage |
| Recovery rate | Failures the run recovered from | Oscillation, not resilience |
| Context exhaustion | Runs hitting the window | The budget cannot finish the task |
| Termination reason | Verified, budget, step cap, error, refusal | Read the distribution, not the mean |
| Cost and latency | Tokens priced, wall clock at p50 and p95 | Quality held, economics regressed |
Read them in pairs. Verification failures beside recovery rate tell you whether the loop closes, which loop structure and stop conditions calls convergence. Steps to completion warns earliest, because degraded reasoning adds attempts before it costs passes.
Record termination reason as a closed enumeration inside the harness, never inferred from text, and store it beside the step count so one group by termination_reason answers how last night’s runs ended. That query is your checkpoint: if unhandled_exception or step_cap appears, name that defect before reading any average.
How do you build a task suite that catches real regressions?
Source tasks from what your agent is actually asked to do, then split capability tests from regression tests, because they deserve different gates.
Mine your episodic logs. Cluster production runs by intent and sample both ends: the intents arriving daily, and the rare ones that cost most when they fail. Every incident becomes a task.
A capability test asks whether the agent can do a class of work at all, and it moves a dashboard number. A regression test asserts that one behaviour worked at one commit and must keep working, and it blocks a merge. Mixed together they give you a suite that is permanently red, which teams ignore, or permanently green, which teaches nothing.
Seed difficulty in tiers so an aggregate pass rate cannot hide a collapse in the hard work.
- Tier 1: one tool call and one verifiable write.
- Tier 2: several steps with a real dependency, so ordering matters.
- Tier 3: a seeded failure to recover from, such as a tool returning 500 on first call.
- Tier 4: under-specified input, where the correct move may be to ask rather than act.
Then write tasks that should fail, because a suite of achievable work rewards an agent that always acts. Include requests needing a permission the agent lacks, records that do not exist, and instructions conflicting with policy. Grade whether it declined, and whether the escalation gave a human enough context to act.
Sample runs for human review on a schedule, weighted toward passes with odd shapes. A pass taking eleven steps where four is normal usually hides a defect behind a correct answer, and each finding becomes a task.
How do you make an agent run repeatable when tools touch the world?
Freeze everything the agent can observe, so the model is the only variable left.
Record and replay tool responses, keyed by tool name plus a hash of the arguments. A call the fixture lacks is itself a finding, because the agent left the expected path. Send writes to an ephemeral database and to stubs for anything that charges money, then reset it between tasks.
Freeze memory as part of the fixture, the step most teams skip. Semantic and procedural memory is an input, so an agent whose memory grew overnight is a different agent this morning. Pin a snapshot per task and load it read-only, or you will measure memory drift and call it model quality. Freeze the clock too, since “today” in a system prompt changes plans.
The model still varies. Temperature zero narrows the output distribution without collapsing it, because hosted inference batches requests and providers update models behind a stable name. Take several samples per task and report the pass rate across them. A task failing every sample is a defect; a task passing sometimes is a reliability problem.
Store each task as a file naming four things: the fixture it replays, the budget it may spend, the assertions that decide it, and how many samples to run. A golden run is the stored trajectory a human accepted as correct, and diffing against it turns “what changed” into a step number.
How do you turn a fuzzy task into a checkable assertion?
Rewrite the task until success is a state you can query, then assert on that state and on what should not have changed.
| Fuzzy task | Assertion the harness can check |
|---|---|
| Summarise the incident and tell the team | A message in #ops-review carries the incident id, and the record’s status equals reviewed |
| Clean up the duplicate customer records | select count() from customers group by email having count() > 1 returns no rows |
| Research the vendor and write a brief | The file exists at the expected path and every citation url returns 200 |
| Fix the failing test | pytest tests/test_billing.py exits 0 and git diff --name-only touches nothing outside src/billing/ |
Every row has two halves. The first asserts what should have changed, the second constrains blast radius, which catches an agent that quietly did something else.
Some properties resist assertion: tone, judgement, whether a summary emphasised the right risk. Extract the checkable parts first, then send only the remainder to a judge.
When is an LLM judge legitimate, and how does it fail?
A model judge is legitimate when the property is genuinely subjective, the rubric is decomposed into specific questions, and the judge has been calibrated against human labels you keep.
LLM-as-judge is the practice of using a language model to score another model’s output against a rubric. It earns three jobs: ranking two candidates on one dimension, checking whether a required element appears in prose, and triaging runs no human has time to read. It never earns the merge gate.
Four biases are well enough known to design against.
- Position bias: the same pair draws different verdicts depending on which candidate appears first. Run both orders, keep the agreements, count the rest as ties.
- Verbosity bias: longer and more assertive answers score higher regardless of correctness. Compare within a length band and penalise unsupported claims.
- Self-preference: a judge scores text from its own model family more generously. Use a different family, or two judges with disagreements routed to a human. If the judge is the model under test, state that beside the number.
- Leniency drift: scores creep upward while nothing improved, because the rubric, the judge prompt, or the judge’s version changed. Pin the version, keep frozen human-labelled examples, and publish the agreement rate beside the score.
Two structural changes beat prompt tuning. Decompose the rubric into yes or no questions about single properties, rather than “rate this one to five”. Then require the judge to quote the span justifying each verdict, and discard verdicts it cannot support.
How do you tell whether an agent’s confidence is calibrated?
Bucket runs by the confidence the agent stated, then compare each bucket’s stated confidence against the pass rate observed inside it.
Require confidence as a typed field in the terminal output rather than a sentence, which is one more reason structured output for LLM systems treats schemas as infrastructure. A calibrated agent’s high bucket passes about as often as it claims. An overconfident agent’s high bucket passes less often, and those are exactly the runs nobody inspects. Expected calibration error is the size-weighted average gap between stated confidence and observed accuracy.
A well-calibrated agent that abstains beats a confident one that never does. An abstention costs a known amount, a human’s attention. A confident error costs an unknown amount, discovered later, after it has propagated into other systems.
Report accuracy against coverage: at each coverage level, the accuracy of the answers the agent chose to give. If accuracy climbs steeply as coverage falls, the confidence signal carries information and a threshold will work. If accuracy stays flat, that value is noise. Publish three numbers: pass rate on attempted tasks, abstention rate, and the rate of runs that were wrong and confident.
What should an agent evaluation harness block in CI?
Block on regression against the base commit, never on an absolute score, because a fixed threshold blocks everything or nothing within a month of being set.
- Any regression task that passed on the base commit and fails on the head, named by task id.
- Any negative task that now acts instead of declining. A lost refusal deserves the weight of a crash.
- A drop in tool-call validity rate, or any new
unhandled_exceptiontermination. - A rise in median cost per task or p95 latency beyond an agreed band, since economic regressions otherwise ship in silence.
Report capability pass rate, judged scores, and calibration curves without blocking on them, since they move for reasons unrelated to this pull request.
When the suite outgrows the merge budget, sample deliberately. Always run every negative task and every task derived from an incident, stratify the remainder across tiers, and select it deterministically from the commit hash so coverage accumulates across merges. Sampling at random makes a flaky failure indistinguishable from an unlucky draw. Quarantine a flaky task with an owner and a date, because deleting them is how a suite becomes green and meaningless.
Store every run with the commit, the model version, the prompt version, and the fixture version. Without the model version you cannot separate a provider-side update from your own regression, and a pull request that removes a tool’s argument validation should fail with a named task id rather than lower an average nobody would question.
Where this goes wrong
The suite drifts away from production. Pass rate climbs while incidents hold steady, because nobody added tasks and tuning fitted the fixtures. Keep a holdout set you never inspect while developing.
Assertions read the transcript instead of the world. A broken integration passes for weeks because the check searched the agent’s summary for the word “created”. Lint the suite for assertions that never touch external state.
A judged score becomes the gate. Someone upgrades the judge and historical numbers turn incomparable. Pin the judge and keep human labels beside every judged score.
The suite gets too slow to gate. Someone marks the job non-blocking. Keep a fast subset on merges and the expensive tail nightly.
Common questions
How many tasks does an agent evaluation suite need?
Enough that every difficulty tier and every negative case has more than one representative. Grow it from evidence rather than imagination: production intents visible in the logs, plus one task per closed incident. A small suite with verifiable assertions beats a large one graded on text.
Can I use a public benchmark instead of building my own suite?
Public benchmarks help you shortlist models and cannot tell you whether your agent works, because your tools, prompts, and data are not in them. Use them for model selection, then gate on your own suite. What they miss is a harness defect.
Does setting temperature to zero make agent runs reproducible?
No. It narrows variation without eliminating it, since hosted inference batches requests and providers update models behind a stable name. Take several samples per task and treat a single passing run as an untested claim.
What is the difference between an evaluation harness and observability?
Observability tells you what happened in production, and evaluation tells you what will happen before you ship. Share one trajectory format between them, because two schemas means two pipelines and confidence in neither.
Next steps
Instrument termination reason and tool-call validity on your existing agent before writing a single task file. Those two usually locate the largest defect you have, and they cost one enumeration and one counter.
Then read the failure taxonomy for production agents, because knowing how runs end is only useful once each ending has a name, a detection signal, and a recovery pattern. Add cost and latency engineering for LLM systems once the suite is stable, since a cost regression is the one a quality gate will not catch.
We publish what we test, including the runs that failed, in AI Fieldwork.