Production evals are three systems, not one
Treating evals as one pipeline is why they're slow and expensive. Splitting deterministic checks, LLM rubric judges, and multimodal extraction into three systems is what I run in production.
“Evals” gets talked about like it’s one thing: you write some test cases, run them against your system, get a pass rate. In practice, once an AI system is actually in production, that single pipeline falls apart, because the things worth checking are not all the same kind of claim. Some are objectively true or false. Some are a matter of judgment. Some aren’t even text yet.
This is the pattern I actually run in production, not a universal framework, just what’s worked for the systems I’ve shipped. I split evals into three separate systems, because conflating them is what makes eval suites slow, expensive, and confusing to debug.
System 1: deterministic checks, in plain code
If a question has a verifiably correct answer, does the output parse as valid JSON, does the returned ID exist in the database, is the total within 2% of the expected value, did the function call use the right tool name, you check it with code. No model involved, no judgment required.
This is the cheapest, fastest, most reliable tier, and it should cover as much ground as possible. The rule I hold myself to: if plain code can decide it, I don’t pay a model to decide it. Every check that moves from code into an LLM judge is a check that got slower, more expensive, and slightly less reliable for no reason.
In my pipelines, this is the only tier that hard-gates CI. If a deterministic check fails, the build fails. Nothing merges.
System 2: narrow rubric LLM judges
Some outputs genuinely require judgment, tone, helpfulness, whether a
summary captured the important point, whether a response followed a style
guide. You can’t write an if statement for “does this sound too formal for
our support chat.” This is where an LLM judge earns its cost, but only if you
keep its job narrow.
A rubric judge should be scoring one specific, well-defined dimension at a time, with a rubric specific enough that two different judge runs mostly agree with each other. “Rate this response 1-10” is not a rubric. “Does this response acknowledge the customer’s stated problem before offering a solution, yes or no” is closer to one.
These judges inform review and can block a merge if a human has set that policy, but in my setup they don’t auto-gate CI the way deterministic checks do, a judge call can drift, get a bad day, or disagree with itself across runs in ways a parser never will.
System 3: multimodal, extract-first
The mistake I see most often with multimodal evals, checking a generated video, a scanned document, a voice response, is feeding the raw media straight into a model and asking it to judge quality directly. That’s expensive, slow, and the model’s attention gets spent on perception instead of judgment.
The pattern that works better: extract first, judge second. Run OCR on
the document, transcribe the audio, pull frames and run ffmpeg to get
structured output, then hand that structured, text-based evidence to a
judge. The judge’s job becomes “does this transcript mention the refund
policy” instead of “listen to this audio and tell me if it mentions the
refund policy.” Same question, radically cheaper and more reliable to answer
once you’ve done the extraction step separately.
The pattern as a table
| Deterministic | LLM rubric | Multimodal | |
|---|---|---|---|
| What it checks | Facts, structure, exact values | Subjective quality on a narrow dimension | Non-text evidence, extracted first |
| Who decides | Plain code | A narrow-scope LLM judge | Extraction pipeline + judge |
| Speed/cost | Cheapest, fastest | Moderate | Highest, unless extraction is cached |
| Gates CI | Yes, always | Only if your team sets that policy | Rarely, usually review-only |
Why the split matters
Treating these as one pipeline tends to produce one of two failure modes: either everything gets routed through an LLM judge because it’s the most flexible tool, which makes the suite slow and expensive and non-deterministic where it shouldn’t be, or everything gets forced into deterministic checks, which means subjective quality dimensions just don’t get checked at all.
Splitting them forces a useful discipline: for every new eval, the first question is “which of these three systems does this belong to,” not “how do I write a prompt for this.” Most of the time, more of your evals belong in system 1 than you’d initially guess.