Building AI Evals That Actually Matter

Eval is the new fancy word on the block. It might sound fancy, but it is indeed very important when it comes to the world of AI and AI systems.

Eval, in simple terms, is QA / Testing for AI. This includes testing models, agents, and AI systems.

The difficult part, however, is building evals that actually matter.

I've been building evals for AI agents recently, and I'd like to share a few principles / philosophies that have helped me build evals that actually move the needle.

When people approach evals, they're often approaching the wrong thing. They try to test the agent behaviour, tool calls, handoffs, etc.

One needs to understand that AI agents / models are built to perform towards a business outcome. Hence, your eval must first validate the end business outcome.

Cookbook I'm following for building effective evals

1. Always have a subject matter expert in the loop

Evals are essentially a set of test scenarios against which our models / agents are evaluated. These scenarios must be carefully curated with people who are actually involved in the specific job that the agent will perform.

For instance, if there is a GTM agent, we must have a GTM leader involved in the golden dataset curation, and the process after.

2. Curate rubrics that are aligned with the end business goal

Generic rubrics for evals aren't very useful.

What makes an eval really useful is when it acts as a release gate, and is successfully able to assess whether the model / agent performed the task it was supposed to do correctly, and whether the user's query actually got resolved.

Have deeper and more specific goals.

3. Stick to deterministic assertions whenever possible

Model inference, as we all know, is non-deterministic. So when an agent is performing, it is naturally bound to have a lot of non-deterministic factors associated with it.

To build an eval that is both effective and stable over the long term, we need to stick to deterministic assertions wherever possible.

For example, a user sends a message to a GTM agent:

Send a message on Day 1, wait for 5 days, and then send a follow-up message.

Here, the value 5 becomes a deterministic assertion that can be easily validated with code.

Similarly, if the final output of your agent can be validated using code. For example, if the final output is a JSON, validate it using code.

Code should verify facts and non-changing things. Models (LLM-as-a-judge) should judge semantic meaning and decision-making.

4. Hybrid approach - Deterministic + Non-deterministic validations

A strong eval validates both the deterministic and non-deterministic aspects of agent performance. For a single execution, have graders that evaluate both.

5. Account for different user personas in the golden dataset

If the agent has a chat-based interface, not all users chat in the same manner. Some users are specific, while others are vague.

Since models are probabilistic, their output for differently worded queries might be completely different, even though both queries are trying to achieve the same semantic outcome from the agent.

Have tests that are able to simulate and validate different customer personas.

6. Trace collection

For an eval to perform deep validation, and not just surface-level scraping, it is very important to fetch and take the agent trace into consideration.

Having access to the agent trace enables us to look into the mechanics of how the agent performed. Which tools did it call? Did those tool calls make sense? Was the state consistent throughout the execution?

As a part of evaluation, the agent trace for a run must also be deeply evaluated.

7. Red-team your agent

Don't just evaluate your agent for the happy path. Try to break it.

Give it incomplete information. Give it conflicting instructions. Ask vague questions. Throw edge cases at it. These are often the kinds of scenarios that expose weaknesses in the agent.

A good agent shouldn't confidently make things up when it doesn't have enough information. It should ask follow-up questions when needed, recover gracefully from bad inputs, and know when to say it cannot perform a task.

The more ways you can try to break your agent during evaluation, the fewer surprises you'll have in production.