A practical guide to evals for AI features

Evals help move beyond "it seems better" when an AI feature changes. They do not guarantee absolute quality, but they make clearer what is being evaluated, which cases matter, and where a new version improved or regressed.

Context

In the previous guides in this series, the goal was to build a practical foundation for thinking about AI applications.

First, the focus was RAG: when an application needs to retrieve external context before answering. Then the focus was chunking, embeddings, and retrieval: how to prepare, split, index, and retrieve that context.

Then came structured outputs: when model output needs to become validatable data for the system. But a question appears right after that: how do you know whether the feature actually improved?

That question catches many people off guard. You adjust a prompt, switch models, change chunk size, alter a schema, add a fallback, or refine an instruction. The answer looks better in a manual test. The first impression is good.

But a good answer in one isolated conversation does not prove the workflow improved. Maybe that specific case got better, while another important case got worse.

The problem

The central question of this guide is: how can you evaluate AI features without depending only on subjective impressions?

When a traditional feature changes, there are usually familiar supports: automated tests, acceptance criteria, code review, QA, logs, metrics, and manual validation.

With AI, part of that still applies. But an extra difficulty appears: the output can vary, quality is not always binary, and not every expected behavior fits into `true` or `false`.

• An answer can be correct but incomplete.
• It can be well formatted but not very useful.
• It can be useful but outside the schema.
• It can be faithful to context but too long.
• It can seem better for one case and worse for another.

If the only validation is "I tested it here and liked it", the process becomes fragile. Not because manual testing is useless. It is important. The problem is when it is the only criterion.

Mental model

1test cases2  -> evaluation criteria3  -> execution4  -> comparison5  -> decision

Test cases represent situations the feature needs to handle. Criteria define what will be observed in each answer. Execution runs one or more versions on the same cases. Comparison shows where things improved, worsened, or stayed the same.

A simple eval can be a table with ten examples, a short rubric, and honest human review. The benefit already appears when the same cases are used to compare different versions.

Evals are not just benchmarks

When people talk about evaluating AI, it is easy to think about public model benchmarks. That is not the focus here.

Benchmarks try to compare models on broader tasks, often with standardized datasets. That can be useful for following the market, but it does not answer a product or engineering question by itself: did this specific feature improve for my workflow?

• Does the summary preserve the important points?
• Does the classification respect the defined categories?
• Does the answer use only the retrieved context?
• Does the JSON follow the expected schema?
• Does the fallback appear when it should?
• Does the output help the next system step?

When to use evals

Evals help mainly when a change can affect behavior in a way that is hard to notice in a single test.

• When changing an important prompt.
• When switching or comparing models.
• When adjusting chunking, embeddings, or retrieval.
• When changing structured output criteria.
• When adding a fallback step.
• When creating automatic classification.
• When generating summaries, extraction, or action suggestions.
• When investigating inconsistent answers.

The point is not to turn every experiment into a heavy process. The point is recognizing when regression risk starts to justify a more organized evaluation.

How to start small

A good start is to build a small set of representative cases. It does not need to be perfect. It needs to be useful.

Imagine a fictional feature that receives support messages and generates a summary with priority, category, and suggested next steps.

11. User cannot access the account.22. User reports a duplicate charge.33. User wants to cancel the subscription.44. User sends praise about a feature.55. User mixes two questions in the same message.66. User writes an incomplete message.77. User includes sensitive data that should not be repeated.88. User asks for something outside the product scope.

These cases do not cover the whole world, but they already help move beyond testing with one comfortable message. Whenever the prompt, model, or workflow changes, the same cases can be used again.

Criteria before the answer

A common mistake is evaluating the answer only after it appears. The problem is that when an answer is convincing, we tend to adapt the criterion to like it.

• Context adherence: does the answer use only present or retrieved information?
• Completeness: were the important points preserved?
• Format: does the output respect the expected schema or structure?
• Usefulness: does the answer help the next workflow step?
• Safety: does the answer avoid exposing unnecessary sensitive data?
• Consistency: do similar cases receive similar treatment?

These criteria can be exact, approximate, or evaluated by rubric. In many cases, a simple scale like `passed`, `failed`, and `needs review` already helps.

Practical example

Let us imagine a fictional input for a triage feature.

1I cannot access my account since yesterday.2I tried to reset the password, but the email does not arrive.3I need to get in today because I have a meeting.

The feature needs to generate structured triage output, with category, priority, summary, suggested actions, and an indication of human review.

1{2  "category": "login",3  "priority": "high",4  "summary": "The user cannot access the account and does not receive the password reset email.",5  "suggested_actions": [6    "Check password reset email delivery",7    "Verify whether there is a recent login block or failure",8    "Route to human support if access is urgent"9  ],10  "needs_human_review": true11}

Now imagine two prompt versions. Version A classifies the case as medium priority, writes a generic summary, and does not request human review. Version B identifies urgency, preserves the email detail, and routes it for review.

By impression alone, version B seems better. But the eval makes the reason clearer: it got category, priority, summary, next steps, human review, and format right.

The eval does not say version B is perfect. It shows that for this case and these criteria, it behaved better. The same test needs to be repeated on other cases to reveal regressions.

Regression is where evals show value

An AI change rarely improves everything at once. Sometimes a new prompt solves one problem and creates another.

• Before, the model forgot required fields; now it respects the schema, but the summary became too generic.
• Before, it answered with more detail; now it is more concise, but omitted important caveats.
• Before, retrieval brought broad context; now it is more precise, but lost complementary information.

Without a set of evals, these regressions appear late. With evals, the question changes from "is this answer good?" to "did this change improve the set of cases that matters?".

Manual testing, checklist, and repeatable eval

Exploratory manual testing is when you talk to the feature, try free examples, and observe behavior. A checklist is when you review fixed points: format, tone, links, sensitive data, context usage, and fallback. A repeatable eval is when you use the same cases and criteria to compare versions.

All three have value. The mistake is treating a happy manual test as sufficient validation for an entire feature.

Where human review fits

Not every criterion needs to be automatic at the beginning. For many projects, the first version of an eval can be manual, especially when criteria have nuance.

• Was the answer useful?
• Did the summary preserve the meaning?
• Is the tone appropriate?
• Was the explanation clear?
• Does the suggestion make sense for a person to review?

Automating a poor evaluation only makes the mistake faster. Before thinking about sophisticated metrics, it is worth understanding what a good answer means in that workflow.

How to compare versions

An eval becomes more useful when it compares versions in a controlled way. Ideally, change one thing at a time when possible: prompt A versus prompt B, current model versus new model, old retrieval versus adjusted retrieval, previous schema versus new schema.

1Date: 2026-07-132Change: triage prompt v23Goal: improve urgency detection4Cases evaluated: 125Overall result: improved urgency in 4 cases, maintained 6, reduced completeness in 26Decision: adjust prompt before approval7Note: the new version tends to over-summarize long messages

This kind of note prevents losing the reasoning behind the decision and helps when the change needs to be reviewed later.

Limits and caveats

Evals help, but they do not solve everything. They do not guarantee that AI will never make mistakes, do not replace human review in sensitive flows, and do not prove that a feature is good for every possible case.

They can also create a false sense of safety if the cases are weak. An eval with overly easy examples can approve a bad change. An eval with vague criteria can become just an opinion with a table.

The question is not "how do I create the perfect eval?". A better question is: what minimum set of cases and criteria would already make this decision less subjective?

Lessons learned

• Evals help compare versions of AI features, not only isolated models.
• A good answer in a manual test does not prove the whole workflow improved.
• Representative cases matter more than a huge list of random examples.
• Criteria need to come before evaluation, not after a convincing answer.
• Regression is one of the main reasons to maintain evals.
• Human review remains part of the process.
• A simple, manual, repeatable eval can already improve the quality of technical decisions.

Conclusion

Working with AI in applications requires a shift in posture. It is not enough to ask whether the answer looks good. You need to ask whether it remains useful, faithful to context, compatible with the expected format, and appropriate for the workflow where it will be used.

Evals fit exactly there. They do not make AI perfect and do not turn technical judgment into pure math. But they help move beyond improvisation.

With a few cases, clear criteria, and version comparison, you can already see better where a change helped, where it introduced regression, and where it still needs review.

In the end, measuring better is not about absolute certainty. It is about making decisions with less guessing.