Why production agent evals beat one-time pilot tests

This week's headlines touched continuous evaluation, multi-agent supervisors, routing for cost, and employee copilot policies. The operational thread is the same: agents change behavior when models, prompts, or tools change. One successful demo is not a quality program.

Here is a practical guide to production agent evaluations for a mid-size operator who needs confidence without a dedicated ML platform team.

Why traditional UAT breaks

Classic user acceptance testing assumes deterministic software: same input, same output. Agents violate that assumption. Temperature, retrieval results, and tool latency all shift answers. A checklist that says "must return exactly this string" will fail good workflows and pass bad ones.

Buyers in 2026 increasingly ask for success-rate bands and re-acceptance after model updates. That is evaluation design, not pessimism.

What a golden dataset is

A golden dataset is a fixed set of realistic tasks with clear success criteria:

• Input: user message plus any required context fixtures.
• Expected behavior: tool called (or not), fields populated, tone, policy compliance.
• Scoring: automated checks where possible, human rubric where not.
• Pass threshold: e.g. 9/10 tasks pass before promote to prod.

Start with ten tasks covering happy path, refusal cases, and one adversarial prompt. Grow monthly as you learn failure modes in production.

Automated vs human scoring

Automate when you can verify structure: JSON schema, required disclaimer, forbidden tools, maximum row count, regex on invoice totals.

Use humans for subjective quality: tone, persuasion, factual nuance, brand voice. Sample 5-10% of production traffic for ongoing review instead of scoring every run.

Hybrid scoring keeps CI fast and catches drift humans notice but scripts miss.

When to run evals

• Pre-merge: prompt or tool schema change in staging.
• Pre-promote: new model version or routing policy.
• Scheduled: nightly regression on production config snapshot.
• Post-incident: add a new golden task that would have caught the bug.

Tie promotion to eval results in your change ticket. "Model bumped" without evals is how quiet regressions reach customers.

Multi-agent and routing evals

If multiple agents collaborate, score the workflow, not each message:

• Did the orchestrator stop within the step cap?
• Was the expensive model invoked only when needed?
• Did the final action match policy (no send without approval)?

Routing policies deserve their own golden tasks: ensure cheap paths handle easy queries and hard queries still escalate correctly.

What to put in a one-page eval policy

1. Owner of the golden dataset and review cadence.
2. Minimum pass rate for production promotion.
3. Escalation when pass rate drops in nightly runs.
4. Human review sampling rate for live traffic.
5. Retention rules for eval logs and production traces.

Common mistakes

• Golden tasks that only cover demos, not edge cases.
• Exact string matching on natural language outputs.
• No evals after vendor model updates you did not initiate.
• Evaluating in dev with different tools than production.

A two-week starter plan

Week 1: Pick one workflow. Document ten tasks. Automate three structural checks. Run baseline on current production.

Week 2: Hook evals to staging deploy. Block promote if pass rate drops more than 10 points from baseline. Add one task from a real support ticket that went wrong.

Bottom line

Production agent evals are how you keep autonomy accountable as models change. Golden datasets, statistical pass bands, and gated promotion turn "it worked in the pilot" into something your board and customers can trust. Pair eval gates with human approval queues for actions that should never ship on a score alone.

Questions on eval design for your agents? Reach out and we will point you at the right pattern.

Matt Potter · Swift Media