Your AI Demo Works. How Do You Know It Still Works Next Week?
A field guide to building an AI evaluation set that catches real failures: test the whole workflow, separate objective checks from judgment calls, and turn traces into regressions you can fix.
By Chakri AI Research
A demo answers one question: can this path work once? An evaluation set answers a more useful one: how often does the system do the right thing across the cases it is meant to handle—and what breaks when we change it? You do not need a large platform to begin. You need examples that resemble actual use and a clear account of what counts as a good result.
Write the success rule before the prompt
Start by naming the task and the acceptable outcome. For a support reply, success might require the correct policy, a direct answer, no invented refund promise, and a handoff when the policy is unclear. For a coding agent, it might mean the requested behavior passes tests, no unrelated files change, and the final report accurately describes what happened.
Build the set from real examples, then add deliberate edge cases: missing fields, contradictory instructions, unusual formats, and requests outside the product’s scope. Remove personal or confidential details before storing examples. Keep a holdout set that you do not repeatedly use to tune the prompt, or you can accidentally optimize for memorizing the test.
Score the parts that fail in different ways
| Dimension | Example check | Scoring method |
|---|---|---|
| Task success | Was the requested record updated correctly? | Deterministic check against the expected state |
| Evidence quality | Does each factual claim have a supporting source? | Human review or a carefully validated rubric grader |
| Tool behavior | Did the agent call the right tool with valid arguments? | Trace checks and schema validation |
| Safety and control | Did it stop before a consequential action needing approval? | Pass/fail policy checks plus adversarial cases |
| User experience | Was the response clear and useful? | Blinded pairwise review or a defined rating rubric |
Use exact assertions wherever possible: valid JSON, expected tool name, correct database state, required citation present. A model grader can help with qualities that are hard to encode as rules, but first compare its judgments with people on a sample. If the grader disagrees with reviewers, investigate the rubric instead of treating its score as ground truth.
Evaluate the path, not just the final sentence
Agent failures often happen between the opening request and the final message: a bad search, a repeated tool call, a missed approval, or an incorrect handoff. OpenAI’s current guide recommends inspecting traces while debugging and moving to datasets and repeated eval runs when you need comparisons over time. Keep those traces scrubbed of secrets and user data that the team does not need.
Anthropic’s agent guidance makes the same engineering point from another direction: start with the simplest workflow that works, measure it, and add agentic complexity only when it improves outcomes. Extra autonomy creates more paths to test and more ways for small errors to compound.
Sources and further reading
Specifications and prices can change. Check the linked pages before making a purchasing or deployment decision.