How to evaluate an AI workflow before it touches production

Most AI workflows are tested the same way: someone tries three examples, they look fine, and it ships. Then a customer sends something odd and the workflow does something nobody expected.

Testing agents is harder than testing normal software because the same input can produce different outputs. It is not impossible, and the practice is simpler than the vocabulary suggests.

Start small and start from real failures#

Anthropic's engineering team, in a post on agent evaluations published on 9 January 2026, recommends starting with about 20 to 50 tasks drawn from real failures and manual checks. Waiting for a large suite only makes evaluations harder to build.

Where do the cases come from?

• Messages and records your team has already handled, with the right outcome known.
• Mistakes you have seen, from support notes or a colleague's "this went wrong last week".
• Edge cases you can think of: empty input, very long input, another language, an angry tone, a request that should be refused.

Write the expected outcome for each. If two people in your team would disagree on whether an output passes, the case is ambiguous. The same post says tasks should be unambiguous enough that two domain experts would reach the same pass or fail verdict. Fix the case before you test against it.

Include the cases where the agent should not act#

The easiest set to write is "the agent does the thing". The more valuable half is "the agent should not do the thing": the refund over the limit that must go to a person, the message that must not be sent, the question with no answer in the knowledge base.

Anthropic's guidance says to include cases where the behaviour should occur and where it should not, to avoid optimising in one direction. If your set is all positive, a workflow that always acts will score perfectly.

Pick the right kind of check#

The same post describes three kinds of grader:

• Code-based checks are fast, cheap and repeatable: does the output contain a value, is it valid JSON, did the right tool get called. They can be brittle if a correct answer is worded differently from what you expected.
• Model-based checks use a model with a rubric to judge open-ended output. They handle nuance, but results vary between runs, they cost more and they need calibration against human judgement.
• Human review is the gold standard and is slow, so use it sparingly, mostly to calibrate the others.

A sensible mix for a business workflow: code checks for everything that can be checked mechanically (format, required fields, forbidden phrases), a model check for tone or completeness, and a person for a sample.

In Cadrix, the evaluator types include checks such as Contains Any, Contains All, Does Not Contain, Starts With and Is Valid JSON, along with measures such as total tokens, API latency and output length. LLM-based evaluators cover the judgement calls. Pair them with a dataset of inputs and expected outputs.

Run it more than once#

Because the output varies, one pass proves little. The Anthropic post distinguishes pass@k, the chance of at least one success in k attempts, from pass^k, the chance that all k attempts succeed. For a customer-facing workflow you care about the second: it should be right every time. The post gives an example: a 75% per-trial success rate gives roughly 42% over three trials. Consistency is harder than a single good run suggests.

So run each case several times, especially the ones where the stakes are high, and look at the spread, not the best result.

Read the transcripts#

A score tells you something went wrong. It does not tell you why. Anthropic's advice is to review the transcripts and grades from many trials, because that is how you tell a real agent mistake from a grader rejecting a valid answer. A near-zero pass rate across many trials usually points to a broken task or grader rather than an incapable agent.

In Cadrix, the evaluation results show each input, the actual output, the expected output and the checks that failed, and the run trace of the workflow shows each step. Read ten failures properly before changing anything.

A step-by-step routine in Cadrix#

1. Build the dataset. Create a dataset of inputs and expected outputs. Start with 20 rows: 12 normal, 4 edge cases, 4 "must refuse or escalate".
2. Define evaluators. Add the code-based checks that apply, and one model-based check for the qualities you cannot check mechanically.
3. Run the evaluation against the workflow. Review the per-row results and the cost and latency figures.
4. Read the failures. For each, decide: workflow problem, instruction problem, missing knowledge, bad test case.
5. Change one thing. A prompt edit, a different model, a new source in the knowledge base.
6. Run again and compare. Keep the old result for reference.
7. Add the surprises. Every real failure after launch becomes a new row in the dataset.

This is the regression loop. Nothing here is exotic. It is a habit.

Test the approval path too#

Evaluation covers what the model produced. It does not by itself confirm that the human steps work. Before release, run the approve path and the reject path in the editor and check what each does. Check what happens if nobody answers. A workflow that is correct but strands its rejections is not ready.

Where the human goes. In two places. A person with domain knowledge writes or checks the expected outcomes, because they are the only one who knows what "right" means. And a person reads the transcripts of failures and a sample of passes, because automated checks can agree with each other and still be wrong.

What to watch for#

• Testing on the cases you tuned on. Keep a few cases you never look at while editing, then run them before release.
• Changing the model without rerunning. A model update or a prompt edit is a change to the system. Rerun the set.
• Treating a high score as a guarantee. Twenty cases do not cover reality. Use the set as a floor, and monitor real runs after launch.
• Letting the checker be the weak point. If a model-based grader is inconsistent, calibrate it on cases a person has already judged.

When is it good enough?#

There is no universal number. Decide in advance, with the people who will live with the outcome, what pass rate and what failure types are acceptable. Zero tolerance for a wrong refund, some tolerance for an awkward sentence. Write the threshold down before you see the results, so it is not negotiated afterwards.

Then ship with an approval step in front of the consequential actions, watch the first weeks closely, and feed what you learn back into the set.

Sources#

• Anthropic Engineering, "Demystifying evals for AI agents", 9 January 2026: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents