🧪 Agent Eval Harness Builder

Eval Harness Design: Customer-Support Triage Agent

Eval Strategy Overview

Three-layer harness with shadow-mode regression detection:

• Unit layer: Tool-selection correctness on 50 cases (does the agent pick lookup_customer_account when account info is needed, etc.)
• Integration layer: Multi-step trajectory scoring on 80 cases (does the agent search KB before drafting, does it verify customer state before personalizing, etc.)
• End-to-end layer: Output quality on 100 golden cases (correct routing + draft accept rate + hallucination check)
• Shadow eval: Live system runs in parallel; eval pipeline diffs against current production for any prompt/model change.

Why this design: your three known failure modes (mis-routing, hallucination, empty drafts) each surface at a different layer. Single-layer evals would miss two of three. With shadow eval added, you catch model-version regressions without waiting for the manual QA cycle.

Golden Test Set

Source distribution: 60% from your existing 'bad cases' Sheet (real failures), 40% from sampled production tickets (success cases for baseline). Total: 230 cases (50 unit + 80 integration + 100 e2e).

E2E sample (10 of 100, illustrative):

Full set of 100 e2e cases mirrors your real category distribution: 30 bug, 25 feature-request, 15 account, 10 billing, 15 integration, 5 other.

Unit layer cases (50):

• Each case is a 1-2 turn excerpt where the agent should pick a specific tool. Score: did it pick the correct tool first?
• Examples: 'Customer asks about plan tier' → first tool should be lookup_customer_account, not search_past_tickets.

Integration layer cases (80):

• Each case is a multi-turn trajectory. Score the trajectory pattern, not the output.
• Examples: 'Did the agent search KB before drafting a technical answer?' (yes/no), 'Did the agent verify account before quoting plan-specific info?' (yes/no).

Trajectory Evaluator

Programmatic checks (cheap, deterministic):

• Tool-call count per ticket. Median + p95. Regression: median doubles or p95 triples.
• Tool-call sequence pattern matching. Define expected sequences per category. Score % of trajectories matching ≥1 valid sequence.
• KB-link extraction from drafts → verify links exist (404 check via cron). Hallucinated KB links are a critical regression.
• Latency per trajectory. Target: p95 <60s, p99 <120s.
• Cost per trajectory. Target: median <$0.04, p95 <$0.08.

LLM-as-judge checks (use Claude Opus, NOT same model as production Sonnet):

• Trajectory reasonableness: 'Did the agent take a reasonable path to the answer?' Score 1-5 with rubric. Calibrated against 30 manually-scored trajectories.
• Hallucination check: 'Does the draft contain any factual claim not supported by tool outputs?' Binary. This is the highest-stakes check.
• Routing-correct check: agreement between agent's category and ground-truth category from human review.

Output Evaluator

Routing accuracy (programmatic): Compare agent's category to ground-truth from human-team feedback. Target ≥92%, alarm at <88%.

Draft quality (LLM-as-judge with Claude Opus):

Prompt: 'You are evaluating a customer-support triage draft. Rate 1-5 on: (a) addresses the customer's actual question, (b) tone-appropriate, (c) factually grounded in tool outputs, (d) actionable next step. Output JSON: {addresses: 1-5, tone: 1-5, factual: 1-5, actionable: 1-5, overall: 1-5, would_a_human_send_this_with_minor_edits: bool}.' Calibrate against 50 human-reviewed drafts.

Hallucination rate (programmatic + LLM):

• Programmatic: extract any quoted numbers, account-IDs, KB-URLs from draft. Verify against tool outputs.
• LLM: 'Does the draft contain any factual claim NOT in the tool outputs above?' Run on every eval trajectory.
• Target: <2% hallucination rate. Alarm at >3%.

Empty-draft detection (programmatic): any draft <30 characters or matches templates like 'I'll get back to you' is flagged. Target: <0.5%.

Cost & Latency Tracking

Track per-trajectory:

• Tokens in / tokens out / total cost
• Wall-clock latency from ticket arrival to draft+route
• Tool-call count
• Number of LLM turns

Dashboard: weekly aggregates with p50/p95/p99. Alarm thresholds:

• Median cost increases >20% week-over-week
• p95 latency exceeds budget for >5% of trajectories
• Tool-call count median increases >50% (signals tool-misselection-then-recovery patterns)

Regression Alarm Setup

On every prompt edit, model change, or tool definition change: automated eval run on the 230-case golden set. Block deploy if:

• Routing accuracy drops >2 percentage points vs baseline
• Hallucination rate increases >1 percentage point
• Median cost increases >20%
• Trajectory-reasonableness score drops >0.3 points (on 5-point scale)

Weekly hygiene run on a fresh sample of 200 production trajectories (sampled, anonymized). Catches drift not visible in fixed golden set.

Distinguishing noise from regression: all metrics tracked with 7-day rolling windows. A single bad day can be sampling variance; a 3-day-rolling drop of 2+ points is signal.

Eval Cadence

• Pre-deploy of any prompt/tool/model change: full 230-case golden run + go/no-go criteria
• Weekly: 200-case fresh production sample + drift dashboard
• Monthly: review and refresh 10-20% of golden cases (rotate stale ones, add new failure modes)
• Quarterly: full audit of eval harness itself — are you measuring what matters?

Implementation Skeleton

/evals
  /golden_cases.jsonl       (230 cases with ground truth)
  /unit_cases.jsonl         (50 tool-selection cases)
  /trajectory_cases.jsonl   (80 integration cases)
  /e2e_cases.jsonl          (100 end-to-end cases)
/runners
  run_unit_evals.py         (replays unit cases, scores tool selection)
  run_trajectory_evals.py   (replays integration cases, scores path pattern)
  run_e2e_evals.py          (full agent invocation, scores output via judge)
  run_shadow_eval.py        (samples live traffic, runs candidate agent in parallel, diffs)
/judges
  judge_routing.py          (programmatic: agent_category vs ground_truth)
  judge_draft.py            (Claude Opus LLM-as-judge for draft quality)
  judge_hallucination.py    (programmatic + LLM hallucination check)
  judge_trajectory.py       (Claude Opus trajectory reasonableness)
/dashboards
  weekly_drift.sql          (rolling 7-day metric query)
  regression_alarm.py       (post-deploy diff vs baseline)

Wire into CI:

• GitHub Action triggered on PR: full golden eval run, post results as PR comment, block merge if regression thresholds tripped.
• Nightly cron: weekly hygiene run.
• Webhook from production: shadow eval continuous run on 5% sampled traffic.

What This Eval Will and Won't Catch

Will catch:

• Routing-accuracy regressions of ≥2 points
• Hallucination-rate increases of ≥1 point
• Cost regressions of ≥20%
• Tool-misselection regressions on the 50 unit cases
• Empty-draft regressions
• Specific known failure modes from your bad-case Sheet

Will NOT catch (known blind spots):

• Long-tail edge cases not in your 230 golden cases (hence weekly fresh-sample run)
• Quality regressions that maintain routing accuracy but produce subtly worse drafts (LLM judge has known calibration limits)
• Drift in customer expectations over time (your bar shifts; eval bar doesn't, until you refresh cases)
• Performance issues caused by the 4 tools' downstream APIs (tool-side problems, not agent problems)
• New failure modes introduced by major architecture changes

Migration Path

1. Week 1: Implement runners + judges. Run baseline eval on current production agent. Freeze numbers.

2. Week 2: Wire CI. Block any PR that regresses >2 points on golden set.

3. Week 3: Implement shadow eval. Run on 1% of traffic.

4. Week 4: Scale shadow to 5% of traffic. Set up dashboards.

5. Week 5: Replace your manual 20-trajectory/week QA with the new harness's weekly hygiene run + a smaller manual review (5/week) of evals-flagged cases.

6. Week 6+: Use the regression alarms to safely scale routing from 60% → 75% → 90% of tickets.

Do NOT increase production routing share until baseline is measured AND CI block is in place. Otherwise you're flying blind on the way up.

Key Takeaways

• Your three known failure modes need three different eval layers (mis-routing → e2e routing-accuracy, hallucination → trajectory + output hallucination check, empty-draft → programmatic length check).
• Use Claude Opus as judge, not Sonnet (your production model). Same-model judging has blind spots on subtle quality regressions.
• Source 60% of golden cases from real failures, not synthetic edge cases. Your bad-case Sheet is gold.
• Implement shadow eval BEFORE scaling routing share. It's the only way to catch regressions that pass golden eval but break on real traffic distribution.
• Cost is part of the eval. A correct-but-expensive agent is a regression; track p50/p95 cost from day one.
• Refresh 10-20% of golden cases monthly. Static eval sets decay as the agent and your customer base evolve.