Evaluating LLM Apps: Metrics That Actually Catch Regressions

Who this is for: engineers and founders who shipped an LLM feature, watched it work great in the demo, then got a Slack message three weeks later saying "the bot said something weird to a customer." If you don't have a repeatable way of evaluating LLM apps before and after every prompt or model change, this guide is for you.

Evaluating LLM apps is nothing like evaluating a REST endpoint. There's no single correct output, the same input can produce different responses on different runs, and "looks right" is not a metric you can put in a CI pipeline. Most teams find out their eval process was inadequate the same week a model provider ships a quiet update and their support bot starts hallucinating refund policies. Below is the framework I use when I build and audit LLM features for clients: what to measure, how to grade it without hiring a QA team, and how to keep watching after launch.

Why accuracy alone lies to you

"Accuracy" implies a single ground truth you can diff against. Most production LLM apps don't have that. A support agent answering "can I get a refund after 45 days?" might give three differently-worded but equally correct answers, and one confidently wrong one. A binary pass/fail accuracy score hides three failure modes that matter more in practice:

• Tone drift - technically correct answer, wrong register (too curt, too apologetic, off-brand).
• Partial correctness - right answer, missing a required disclaimer or a needed follow-up question.
• Latent failure - the model calls the wrong tool, gets a correct-looking result anyway, and nobody notices until the tool call is audited.

None of these show up in a simple exact-match or BLEU-style score. That's why every eval suite worth using scores on multiple axes: correctness, groundedness (is it supported by the retrieved context?), safety, tone, and task completion, each independently. If you're building the retrieval layer that feeds these answers, see rag-for-smb-pdf-to-assistant for how groundedness ties back to your source documents.

The three layers of an eval suite

Think of evaluation as three layers that run at different frequencies and different costs:

1. Deterministic checks (run on every commit, cost ~$0)

Regex and schema validation: does the output parse as valid JSON if you asked for structured output? Does it contain the required disclaimer string? Did it avoid a banned phrase? These are cheap, fast, and catch a surprising number of regressions - a broken JSON schema after a prompt edit is one of the most common silent failures I've seen in the wild. If your app relies on structured output from the model, pair this layer with the patterns in structured-outputs-guide.

2. Model-graded eval (run on every PR, cost ~$0.01-0.10 per test case)

Use a second, usually stronger, model as a judge. You give it the input, the candidate output, and a rubric, and it scores 1-5 or pass/fail with a reason. This scales far better than human review and catches things regex can't - tone, relevance, whether the answer actually addresses the question asked.

3. Human review (run weekly or on flagged samples, cost = your team's time)

Model judges have blind spots correlated with the same model family's blind spots. A human spot-check of 20-50 real production transcripts a week catches what both automated layers miss, and it's your best early signal that the eval rubric itself needs updating.

Tip: don't skip layer 3 because layers 1 and 2 look green. I've seen a client's eval suite score 94% "pass" for two weeks while their actual CSAT dropped, because the rubric was grading against an outdated policy document nobody updated.

Building a golden dataset

Before you can grade anything, you need a set of representative inputs with known-good (or known-bad) outputs. Don't write these from imagination - pull them from real usage:

• 30-50 real user queries from logs, stratified across intents (not just the easy ones).
• 10-15 adversarial cases: prompt injection attempts, off-topic questions, requests for something the bot should refuse.
• 5-10 edge cases specific to your domain: ambiguous dates, multi-part questions, questions in a second language if you support Hebrew and English.

Store this as a versioned file, not a spreadsheet someone edits in place. A JSON or YAML file in the repo, reviewed via pull request, is the right level of ceremony - see eval-driven-ai-development for the full workflow of writing evals before you write the prompt, which is the single highest-leverage habit in this whole discipline.

{
  "id": "refund-policy-45-days",
  "input": "Can I get a refund after 45 days?",
  "category": "policy",
  "expected_facts": ["standard window is 30 days", "case-by-case exceptions require manager approval"],
  "must_not_contain": ["yes, no problem", "full refund guaranteed"],
  "tags": ["refunds", "adversarial-adjacent"]
}

Model-graded evaluation in practice

A judge prompt needs the same rigor as any production prompt. A vague "rate this response 1-10" rubric produces noisy, unreproducible scores. Give the judge model a structured rubric and force structured output back:

JUDGE_PROMPT = """
You are grading a customer support response for a SaaS product.

Question: {question}
Response: {response}
Required facts: {expected_facts}
Forbidden phrases: {must_not_contain}

Score each dimension 1-5:
- correctness: does it state the required facts accurately?
- groundedness: is every claim traceable to the provided context?
- tone: professional, warm, not robotic or dismissive?
- safety: no forbidden phrases, no policy overpromises?

Return JSON: {{"correctness": int, "groundedness": int, "tone": int, "safety": int, "reasoning": str}}
"""

Run this with a model that is NOT the one being tested - grading Claude Sonnet's output with Claude Sonnet as judge tends to inflate scores because the judge shares the same blind spots. A common, cost-effective pattern: use a cheaper or different-family model as first-pass judge, then escalate low scores to a stronger judge or a human.

Warning: judge models are inconsistent run to run. Run each test case through the judge 3 times and take the median score, or you'll chase noise instead of real regressions.

Metrics worth tracking beyond pass/fail

If you're comparing providers or deciding whether a cheaper model is "good enough," run the same golden set against each and compare these metrics side by side rather than trusting published benchmarks - see ai-model-pricing-compared-hebrew for current pricing context and gpt-vs-claude-coding-2026 for a head-to-head on coding-specific tasks, which generalizes to how you'd structure any model comparison.

Regression testing on every prompt or model change

Treat your prompt file the way you treat application code: a change requires a passing eval run before merge. The workflow that actually holds up under real deadlines:

1. Prompt or model change lands in a branch.
2. CI runs the full golden set through the deterministic checks (layer 1) - instant fail on schema breaks.
3. CI runs the model-graded suite (layer 2) and compares scores against the last known-good baseline stored in the repo.
4. Any dimension dropping more than a set threshold (I use 10% relative drop) blocks the merge and requires a human look.
5. On merge, a canary release to 5-10% of traffic runs for 24-48 hours before full rollout.

This is the same discipline as regular software regression testing, just with fuzzier assertions. The teams that skip step 5 are the ones surprised when a provider's silent model update (which happens more often than vendors advertise) changes behavior for everyone at once.

Production monitoring: what to actually watch

Evals catch known failure modes. Monitoring catches the ones you didn't think to test for.

• Sample and log 100% of production transcripts (with PII handling per your privacy policy) - you cannot debug what you didn't record.
• Automated anomaly flags: sudden spike in response length, sudden drop in tool-call rate, spike in refusal rate.
• User feedback loop: a thumbs up/down on every AI response is the cheapest signal you'll ever get, and it should feed directly back into your golden dataset review queue.
• Weekly sample review: pull 20-30 flagged or randomly sampled transcripts and have a human score them against the same rubric the judge model uses. Track judge-vs-human agreement over time; if it drops, your judge prompt needs work, not just your app.

For agent-style apps with multiple tool calls and steps, watch each step's outcome independently rather than only the final answer - a wrong intermediate tool call can still coincidentally produce a plausible final answer. The patterns in agentic-workflow-patterns and llm-tool-calling-guide are useful references for instrumenting each hop.

When to add guardrails instead of more eval

Evaluation tells you something is wrong after the fact. Guardrails stop it before the user sees it. If your eval suite keeps catching the same category of failure release after release, that's a signal to add a runtime guardrail rather than tightening the eval threshold further - a regex filter for banned phrases, a second-pass moderation check, or a hard-coded fallback for known-risky topics. See llm-guardrails-safety for the specific patterns.

A minimal eval stack you can build this week

You don't need a vendor platform to start. A working v1:

• A golden-set.json file in your repo (30-50 cases).
• A Node or Python script that runs each case through your app and stores the raw output.
• A second script that sends output + rubric to a judge model and stores scores in a CSV or SQLite table.
• A GitHub Action that fails the PR if the average score drops below baseline.
• A weekly cron reminder to spot-check 20 real transcripts by hand.

That's it. It costs a few dollars a month in judge-model API calls and a couple of hours to set up. The upgrade path from there - dedicated eval platforms, automated dataset expansion, drift dashboards - is worth it once you're running more than a handful of prompt changes a month, but don't wait for that scale to start.

FAQ

What's the difference between LLM evaluation and traditional software testing?

Traditional testing checks for exact expected output; LLM evaluation checks for a range of acceptable outputs against a rubric, because the same correct answer can be phrased many different ways. You still want deterministic checks where possible (schema validation, banned phrases), but the core of LLM eval is scored, not binary.

How many test cases do I need in a golden dataset?

Start with 30-50 real cases stratified across your main intents, plus 10-15 adversarial cases. That's enough to catch most regressions in CI. Expand it over time by adding every real production failure you find during monitoring - your golden set should grow from actual incidents, not just brainstorming.

Can I use the same model as both the app and the judge?

You can, but scores will run optimistic because the judge shares the same blind spots as the model being graded. Where possible, use a different model family as judge, or at minimum a different model version, and validate judge-vs-human agreement periodically to make sure the judge is actually catching what humans would catch.

How often should I re-run evals in production?

Run the full suite on every prompt or model change before merge, and run a lighter continuous check (sampling live traffic) daily. Providers update models without always announcing behavior changes, so a scheduled daily or weekly re-run against your golden set is the only way to catch drift that isn't tied to your own deploys.

What tools are commonly used for LLM evaluation?

Open-source options include promptfoo and DeepEval for running structured eval suites, and Braintrust or LangSmith for hosted tracing plus eval dashboards if you want less DIY setup. Most teams start with a homemade script and a spreadsheet, then migrate to a platform once volume or team size justifies it.

How is this different from RAG evaluation specifically?

RAG evaluation adds a retrieval-quality dimension on top of everything here - you need to separately score whether the retrieved chunks were relevant and whether the final answer was actually grounded in them, since a bad retrieval can produce a fluent but wrong answer. See rag-for-smb-pdf-to-assistant for the retrieval side specifically.

Get help setting this up

If you're shipping an LLM feature and don't yet have a repeatable way of evaluating LLM apps before every release, it's worth fixing before the next silent regression reaches a customer. Book a free 30-minute call through the contact form or message me directly on wa.me/972585802298 and we'll walk through your current setup.

Sources

• Anthropic - Building Evals
• OpenAI - Evals framework
• promptfoo documentation
• DeepEval documentation
• LangSmith evaluation docs