Why I run evals before every agent deploy
A small, representative eval baseline in CI catches the regressions that manual spot-checking misses. Here is how I size the set and what I actually measure.
The failure mode I keep seeing on agent projects is not a dramatic one. Nobody ships something obviously broken. Someone tweaks a system prompt to fix one annoying behaviour, runs the agent manually two or three times, sees it work, and merges. A week later a different behaviour has quietly degraded, and nobody can say which change did it, because there was never a measurement in between.
That is the whole argument for evals in CI. Not that they prove your agent is good, but that they tell you whether today’s version is worse than yesterday’s on cases you already decided you care about.
The baseline is small on purpose
I aim for something between 20 and 50 cases before an agent goes anywhere near production traffic. That range is not magic. It is the smallest set I have found that is still honest about the system’s actual surface area, and small enough that it runs on every pull request without anyone resenting it.
Under roughly 20 cases, a single flaky run swings the pass rate so hard that the signal is useless and people start ignoring it. Past 50, two things happen: the run gets slow enough that engineers skip it locally, and the set usually stops being curated. It becomes a pile of cases someone added once, including near-duplicates that all fail together and make one regression look like six.
Representative matters more than large. If your agent handles five classes of request, the set should cover all five, including the awkward ones: the ambiguous query, the request where the right answer is “I cannot do that,” the input where a tool is going to return an error. A baseline made entirely of happy paths will stay green through exactly the regressions you most need to catch.
The three things I measure
For an agent specifically (as opposed to a single-shot model call), three dimensions carry most of the weight.
Tool correctness. Did the agent call the tools it should have called, with arguments that make sense, and did it avoid calling the ones it should not have? This is the one that catches prompt edits most reliably, because tool selection is extremely sensitive to wording. It is also the cheapest to check, since the expected tool trace is usually something you can assert against directly rather than needing a judge model.
Task completion. Did the run actually accomplish what was asked, end to end? Tool correctness can pass while completion fails: the agent picks the right tools, gets a result, and then stops one step early or summarises something it never retrieved. Completion is the metric that notices when the agent is going through the motions.
Answer relevancy. Is the final output responsive to what the user actually asked? This catches a specific and common drift where the agent produces something true, well-formatted, and about a slightly different question than the one posed. It usually needs an LLM judge, which means it is the noisiest of the three and the one I treat as a trend rather than a hard gate.
That split maps onto something I wrote about separately in production evals are three systems, not one: deterministic assertions, judged scores, and everything else are different machines with different costs, and bolting them into one pipeline is how eval suites get slow and expensive.
Gating, not just reporting
A baseline that only prints numbers gets ignored within a month. The version that works has the deterministic checks as a hard gate (tool correctness dropping below the recorded baseline fails the build) and the judged metrics as a reported trend with a wider threshold. That asymmetry is deliberate. Failing a PR on a judge model’s half-point movement teaches people to rerun CI until it passes, which is worse than having no gate at all.
I also keep the baseline numbers committed in the repo rather than inferred from the last run. When someone improves the agent and the numbers go up, the new floor is a code change that gets reviewed like any other.
About the tooling
Two things I have reached for while setting this up: DeepEval’s metrics documentation is a useful starting point for how the common agent metrics (including tool correctness, task completion, and answer relevancy) are defined and scored, and Langfuse’s evaluation docs cover running and tracking evaluations against traced runs.
I want to be careful here. These are starting points, not endorsements, and certainly not a claim that either is the right answer for your stack. The eval tooling space moves quickly and most teams end up with something partly custom, because the hard part is never the metric implementation. The hard part is deciding which 30 cases genuinely represent your product, and keeping that set honest as the product changes.
The tool you pick matters much less than whether the suite runs automatically and whether anyone looks at it when it goes red.