Article·8 min read

Your AI Demo Works. How Do You Know It Still Works Next Week?

A field guide to building an AI evaluation set that catches real failures: test the whole workflow, separate objective checks from judgment calls, and turn traces into regressions you can fix.

By Chakri AI Research

Your AI Demo Works. How Do You Know It Still Works Next Week?

A demo answers one question: can this path work once? An evaluation set answers a more useful one: how often does the system do the right thing across the cases it is meant to handle—and what breaks when we change it? You do not need a large platform to begin. You need examples that resemble actual use and a clear account of what counts as a good result.

A recurring evaluation workflow moves from representative cases through system runs and review into a regression set.
Turn failures into future test cases: each workflow change should be checked against the same representative examples.

Write the success rule before the prompt

Start by naming the task and the acceptable outcome. For a support reply, success might require the correct policy, a direct answer, no invented refund promise, and a handoff when the policy is unclear. For a coding agent, it might mean the requested behavior passes tests, no unrelated files change, and the final report accurately describes what happened.

Build the set from real examples, then add deliberate edge cases: missing fields, contradictory instructions, unusual formats, and requests outside the product’s scope. Remove personal or confidential details before storing examples. Keep a holdout set that you do not repeatedly use to tune the prompt, or you can accidentally optimize for memorizing the test.

Score the parts that fail in different ways

A practical evaluation scorecard
DimensionExample checkScoring method
Task successWas the requested record updated correctly?Deterministic check against the expected state
Evidence qualityDoes each factual claim have a supporting source?Human review or a carefully validated rubric grader
Tool behaviorDid the agent call the right tool with valid arguments?Trace checks and schema validation
Safety and controlDid it stop before a consequential action needing approval?Pass/fail policy checks plus adversarial cases
User experienceWas the response clear and useful?Blinded pairwise review or a defined rating rubric

Use exact assertions wherever possible: valid JSON, expected tool name, correct database state, required citation present. A model grader can help with qualities that are hard to encode as rules, but first compare its judgments with people on a sample. If the grader disagrees with reviewers, investigate the rubric instead of treating its score as ground truth.

Evaluate the path, not just the final sentence

Agent failures often happen between the opening request and the final message: a bad search, a repeated tool call, a missed approval, or an incorrect handoff. OpenAI’s current guide recommends inspecting traces while debugging and moving to datasets and repeated eval runs when you need comparisons over time. Keep those traces scrubbed of secrets and user data that the team does not need.

Anthropic’s agent guidance makes the same engineering point from another direction: start with the simplest workflow that works, measure it, and add agentic complexity only when it improves outcomes. Extra autonomy creates more paths to test and more ways for small errors to compound.

Sources and further reading

Specifications and prices can change. Check the linked pages before making a purchasing or deployment decision.