LLM Evals: Testing AI Features Before Your Users Do

Your users are already testing your AI feature. The only question is whether you see the failures before they do. Most teams ship LLM features with less rigor than a CSS change, and it shows in support tickets, churn, and screenshots on social media.

Why "it looks good" is not a test strategy

Traditional software fails loudly. A broken function throws, a bad deploy 500s, monitoring pages someone. LLM features fail silently. The endpoint returns 200, the response is fluent and confident, and it is wrong. No stack trace. No alert. Just a user who quietly stops trusting your product.

Three properties make LLM features uniquely hard to test:

• Nondeterminism. The same prompt can produce different outputs across runs, even at temperature 0, because of batching and floating point behavior on the provider side.
• Unbounded input space. You cannot enumerate what users will type. Your test cases are always a sample, never a specification.
• Coupled regressions. Change one line of a system prompt to fix case A and you silently break cases B through F. Upgrade the model and everything shifts at once. Prompt changes are global, not local.

Manual spot checking does not survive contact with any of this. You paste five inputs into a playground, they look fine, you ship. Two weeks later a prompt tweak for one complaint has degraded a behavior nobody rechecked. This is the default failure loop, and evals are how you break it.

What an eval actually is

Strip away the vendor marketing and an eval is four things:

• A dataset: input cases, each optionally paired with a reference output or acceptance criteria.
• A task runner: the code that feeds each case through your real pipeline (prompt, retrieval, tools, everything), not a simplified copy of it.
• Scorers: functions that turn each output into a number or a pass/fail.
• A harness: the thing that runs it all, aggregates results, and compares against a baseline.

That is it. An eval is a test suite where assertions are probabilistic and the fixture set is curated from reality. It is not a public benchmark. MMLU scores tell you nothing about whether your refund bot invents refund amounts. Generic benchmarks rank models. Evals test your product.

The three scoring families

Every scorer you will ever write falls into one of three buckets, and picking the wrong one is the most common eval mistake we see.

Deterministic scorers. Exact match, regex, JSON schema validation, "does the SQL execute", "is the cited doc ID actually in the retrieved set". Cheap, fast, zero flakiness. Use these for everything structural. If your feature outputs JSON, schema validation is your first assertion, always, before any quality judgment.

Human labels. The ground truth, and the most expensive. You cannot afford humans on every run, but you need them to build the dataset and to calibrate everything else. A few hours of a domain expert labeling 100 outputs is worth more than any tooling purchase.

LLM as judge. A second model scores the output against a rubric. This is the workhorse for fuzzy qualities: groundedness, tone, helpfulness, refusal correctness. It works, but only under discipline, because judges have known biases:

• Verbosity bias: longer answers score higher regardless of quality.
• Position bias: in pairwise comparisons, the first option wins more often. Swap the order and average.
• Self-preference: models rate their own outputs higher. Judge with a different model family than the one generating.

The mitigation that matters most: calibrate the judge against human labels before trusting it. Take 50 to 100 outputs your team has labeled, run the judge, and measure agreement. In our experience a well-built rubric judge reaches roughly 80 to 90 percent agreement with human raters on a narrow task. Below that, fix the rubric, not the threshold. A judge you never calibrated is a random number generator with good grammar.

Write rubrics as binary criteria, not 1 to 10 scales. "Does the answer cite at least one retrieved document: yes/no" is stable. "Rate groundedness out of 10" drifts run to run and means nothing.

Building your first eval set

Do not start by generating 1,000 synthetic cases. Start with 30 real ones. A small dataset drawn from production traces beats a large one hallucinated by a model, because synthetic data inherits the generator's blind spots, which are usually the same as your feature's blind spots.

Here is the sequence we run on client projects:

1. Pull 200 to 500 real interactions from logs or traces. Before launch, use pilot users or the founder dogfooding, but real humans, not prompts you wrote yourself.
2. Label the failures. Read them. Tag every bad output with a failure mode: hallucinated facts, missed refusal, wrong tone, broken formatting, ignored context.
3. Curate 30 to 50 golden cases covering each failure mode plus the happy paths. Each case gets an input, the context it needs, and explicit pass criteria.
4. Slice by category. An aggregate score hides everything. "87 percent overall" is useless. "100 percent on formatting, 60 percent on refusals" tells you exactly what to fix.
5. Add every production failure as a new case. This is your regression suite. It compounds. Six months in, this dataset is one of the most valuable assets in the codebase.
6. Version the dataset in git next to the prompts it tests. Prompt v12 was evaluated against dataset v3: you want that traceable.

If your feature includes retrieval, eval the retriever separately from the generator. Recall at k and precision on the retrieval step, groundedness on the generation step. When the end-to-end score drops you need to know which half broke. We cover the retrieval side in depth in RAG systems explained.

Tooling: a working setup

You do not need a platform to start. A pytest file and a judge function will carry you surprisingly far:

def judge(rubric: str, input: str, output: str) -> bool:
    resp = client.messages.create(
        model=JUDGE_MODEL,  # different family than the generator
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(
            rubric=rubric, input=input, output=output)}],
    )
    return parse_verdict(resp)  # returns True only on explicit PASS

@pytest.mark.parametrize("case", load_cases("golden/refunds.jsonl"))
def test_refund_answers(case):
    out = run_pipeline(case.input)          # the REAL pipeline
    RefundAnswer.model_validate_json(out)   # structure first, deterministic
    assert judge(case.rubric, case.input, out)

When you outgrow that, promptfoo gives you declarative configs, side-by-side model comparison, and CI integration for free:

# promptfooconfig.yaml
prompts:
  - file://prompts/support_answer.txt
providers:
  - anthropic:claude-sonnet-4-5
tests:
  - vars:
      question: "Can I get a refund after 45 days?"
    assert:
      - type: is-json
      - type: llm-rubric
        value: >
          States the 30 day policy correctly. Does not invent
          exceptions. Offers escalation to a human.
defaultTest:
  assert:
    - type: latency
      threshold: 3000
    - type: cost
      threshold: 0.02

Note the latency and cost assertions. Quality regressions are not the only regressions. A prompt change that doubles token usage is a bug, and your eval harness is the right place to catch it. That budget discipline pairs directly with the techniques in LLM cost optimization.

Hosted platforms (Braintrust, LangSmith, Langfuse) earn their keep once you need shared dashboards, trace-linked eval results, and non-engineers reviewing outputs. Adopt them for collaboration, not because you think you cannot write a for loop.

Wiring evals into CI

An eval you run manually is a demo. An eval in CI is a control. The workflow:

• On every PR that touches prompts, models, or pipeline code, run the golden set and post a comparison against the main branch baseline. Diff the score per slice, not just the aggregate.
• Gate on regression, not perfection. "No slice drops more than 5 points" is a workable gate. "95 percent overall" invites gaming and blocks unrelated work when a flaky case wobbles.
• Run each case multiple times for agentic flows. Single-run pass rates lie about reliability. If a case passes 3 of 5 runs, your users will hit the failing 40 percent. For multi-step agents, score pass^k (all k runs pass), because compounding step failures destroy end-to-end reliability faster than any single metric suggests.
• Keep the CI set small and fast: 30 to 80 cases, minutes not hours. Run the full extended set nightly.

Treat statistical noise honestly. With 40 cases, a 2 point swing is one flipped case. Do not celebrate it, do not revert over it. Rerun before reacting. The mechanics of trustworthy gates are the same ones covered in CI/CD pipelines teams trust: fast, deterministic where possible, and never routinely skipped.

Production: online evals close the loop

Offline evals catch what you thought to test. Production catches the rest.

• Sample live traffic (5 to 10 percent is a typical starting point) and run your cheap scorers plus a judge on the samples, asynchronously, off the request path.
• Track slice scores over time. A slow drift in groundedness usually means your data changed, not your code: new documents, new user segment, new phrasing patterns.
• Instrument implicit feedback: retries, rephrases, thumbs down, human handoffs, task abandonment. These are your highest-signal failure detectors and they cost nothing.
• Route flagged outputs into a review queue, and promote confirmed failures into the golden set. This is the flywheel: production failure becomes eval case becomes permanent regression guard.

This is observability work as much as ML work. Traces that link user input, retrieved context, model output, and scores are the substrate everything else sits on. If that muscle is weak, start with observability: logs, metrics, traces.

Failure modes to avoid

• Overfitting to the eval set. If you iterate on prompts against the same 40 cases for months, you are training on your test set. Hold out a split you only check before release.
• Judging with the generator. Same model family grading its own homework inflates scores. Cross-family judging or bust.
• Scale rubrics. Numeric scales without anchored criteria produce noise. Binary criteria, aggregated, produce signal.
• Evaluating a simplified pipeline. If the eval calls the model directly but production goes through retrieval, tools, and truncation logic, you are testing a different product.
• All aggregate, no slices. The average hides the segment that is on fire.
• No baseline comparison. A score means nothing alone. A delta against main means everything.

A decision framework for scorer choice

Keep it simple:

• Output has verifiable structure or facts (JSON, SQL, IDs, dates, exact policy numbers): deterministic scorer. Never pay a judge for what a regex can check.
• Output quality is fuzzy but describable (grounded, on-tone, correctly refuses): calibrated LLM judge with binary rubric criteria.
• Stakes are high or the rubric will not converge (medical, legal, brand-critical copy): human review, with the judge as a pre-filter to focus human time on likely failures.

Most real features use all three in layers: schema check first, judge second, sampled human audit third.

How Innovation T can help

Innovation T builds LLM features with the eval harness included, because we do not consider an AI feature done until it has a golden dataset, CI gates, and production scoring wired in. We have built this stack for chat assistants, RAG products, and multi-step agents, and we have retrofitted it onto AI features that were already live and misbehaving.

If you are shipping AI on vibes and want to ship it on evidence instead, see our software and AI engineering services or talk to us about an eval audit of your current pipeline. The first version of your golden dataset can exist within a week.