Affiliate disclosure: This post contains affiliate links. If you buy through them, we may earn a commission at no extra cost to you. We only recommend tools we’d run our own business on.
Why most AI support-agent tests are rigged
Here’s how vendor benchmarks usually work: the vendor picks the tickets, runs their own agent, scores their own answers, and publishes the percentage. Every step of that chain is an opportunity to be accidentally (or deliberately) optimistic. Cherry-picked tasks inflate the task set. The scorer knows which answers came from the new model. Costs are quoted as “tokens” instead of dollars per resolved ticket.
- Why most AI support-agent tests are rigged
- The protocol: 30 fixed tickets, blind-scored
- Step 1 — Build the task set from your real tickets
- Step 2 — Freeze the inputs
- Step 3 — Score blind
- Step 4 — The rubric
- Step 5 — The cost-per-resolution math
- The blank scorecard (copy this)
- What the test can’t prove (read this before you buy anything)
- FAQ
- Methodology
- Sources
If you’re evaluating a support agent for your own business — or building one for a client — you need a protocol that removes your own optimism from the equation. This is the lab procedure we recommend. It fits on one page, takes an afternoon, and its biggest feature is telling you what the test can’t prove.
The protocol: 30 fixed tickets, blind-scored
Step 1 — Build the task set from your real tickets
Pull 30 real tickets from your own support history. Anonymize them (strip names, order numbers, addresses), and stratify across categories — 5 tickets each:
| Category | Example ticket |
|---|---|
| Billing / pricing | “I was charged twice this month.” |
| Refunds / cancellations | “How do I cancel before the renewal?” |
| Technical troubleshooting | “The export keeps failing at 60%.” |
| Account access | “Locked out after changing email.” |
| Policy questions | “Do you ship to Canada?” |
| Product / how-to | “How do I add a second user?” |
Thirty is a compromise number: large enough to cover your categories, small enough to score by hand in an afternoon. Write it down as a limitation from the start (more on that below).
Step 2 — Freeze the inputs
Export the same 30 tickets, in the same order, with the same knowledge base and the same tool access, for every agent or configuration you test. Temperature matters: set it low (0.1–0.3) and log the exact setting. If the agent can browse or call tools, that must be identical across runs. Change nothing between runs except the thing you’re testing.
Step 3 — Score blind
This is the step almost nobody does and it changes everything: the scorer must not know which agent produced which answer. Run two agents (or your agent vs your human baseline), strip the outputs of identifying formatting, shuffle the order, then score. It takes ten extra minutes and eliminates the single largest source of bias in self-evaluation.
Step 4 — The rubric
Score each ticket on three dimensions, 0–5 points each. Resolution correctness carries the most weight in the total because that’s what the customer feels.
| Dimension | 0–1 (poor) | 2–3 (acceptable) | 4–5 (good) |
|---|---|---|---|
| Resolution correctness (×2) | Wrong or incomplete answer | Correct but required a follow-up | Correct, complete, first contact |
| Tone & clarity | Rude, confusing, or copy-pasted walls | Polite but unclear or padded | Clear, warm, appropriate length |
| Escalation judgment | Escalated a routine ticket / failed to escalate a serious one | Slightly early or late escalation | Escalated exactly when a human was needed |
Maximum 25 points per ticket (correctness doubled). A ticket is resolved if correctness scores ≥ 4.
Step 5 — The cost-per-resolution math
Vendors quote cost per ticket. That number is meaningless — it includes tickets the agent handled badly. What matters is cost per resolution:
Cost per resolution = (platform cost + API cost) ÷ resolved tickets
Platform cost is your monthly subscription for the agent builder (n8n Cloud, a managed agent platform, whatever you pay) prorated to the test period. API cost is the actual token spend logged during the run — pull it from your provider dashboard, don’t estimate it.
A worked example with clearly-labeled hypothetical round numbers: you run 2,000 tickets in a month through a $500/month platform with $80 of API spend. The agent handles all 2,000, but blind scoring says 1,400 were genuinely resolved. Cost per ticket: ($500 + $80) ÷ 2,000 = $0.29. Cost per resolution: ($500 + $80) ÷ 1,400 = $0.41. The 41-cent number is the one that belongs in your business plan — because the 600 unresolved tickets cost you human time on top of the agent bill.
For the voice layer of a phone agent, ElevenLabs is a solid text-to-speech option (22% recurring commission for 12 months on Starter/Creator/Pro/Scale plans, checked October 2026) — and you should fold its per-minute voice cost into the same formula. [AFFILIATE LINK: elevenlabs]
The blank scorecard (copy this)
## Support agent benchmark scorecard
Date: ______ | Agent/config: ______ | Temperature: ______ | KB version: ______
| # | Category | Correctness (0-5 x2) | Tone (0-5) | Escalation (0-5) | Total (/25) | Resolved? (correctness >=4) |
|---|----------|----------------------|------------|------------------|-------------|-----------------------------|
| 1 | Billing | | | | | |
| 2 | Billing | | | | | |
| ... (30 rows total, 5 per category) ... |
| 30| How-to | | | | | |
Total resolved: __ / 30 | Resolution rate: __ % | Avg score: __ / 25
Escalation misses (should have escalated): __
Unnecessary escalations: __
Platform cost: $______ | API cost: $______ | Cost per resolution: $______What the test can’t prove (read this before you buy anything)
A 30-ticket test does not prove production readiness. State this plainly in your report, to yourself and to any client:
- Sample size. With 30 tickets and an 80% observed resolution rate, the true rate is roughly 66–94% at 95% confidence — a ±14-point band. Two agents scoring 24/30 and 26/30 are not measurably different.
- Adversarial behavior. Real customers are angry, vague, and multi-lingual. Your clean ticket set doesn’t test rage, typos, or the customer who opens with “your product is garbage.”
- Drift. Knowledge bases rot. Model providers update models silently. A benchmark is a snapshot, not a certification — rerun it monthly and after every knowledge-base change.
- Regression blind spots. The agent that scores best on your 30 tickets may have gotten worse on 300 tickets you didn’t include. Keep a small regression set frozen and add new tickets to it over time.
AWL verdict: Run the 30-ticket blind test before you pay for any support-agent platform — it takes one afternoon and it’s the only honest signal you’ll get this side of production. But treat the result as a screening test, not a verdict. An agent that fails here will fail in production; an agent that passes here still has to survive real customers. Budget one full week of shadow mode (agent drafts, human sends) before going live, and keep measuring cost per resolution, never cost per ticket.
If you’re assembling the rest of your AI stack on a budget, start with our AI business stack: 7 tools under $50 — then benchmark them honestly with this protocol.
FAQ
How many tickets do I really need?
30 is the minimum for a useful hand-scored test covering six categories. Below 20, the confidence band is so wide the result is decoration. Above 100, scoring by hand becomes a project — at that point sample randomly from your real queue and score a rotating subset weekly.
Can I use AI to score the agent’s answers?
You can, for a first pass — but an LLM judge grades against its own biases and tends to prefer longer, more confident-sounding answers. If you use one, spot-check 20% of its scores by hand. For purchase decisions, human blind scoring is the standard.
Should I include my own tickets or use a standard benchmark?
Your own, anonymized. Public support benchmarks test generic chatbot skills; your business has specific policies, edge cases, and tone requirements. A generic benchmark passing means nothing for your refund policy.
How do I handle PII in real tickets?
Anonymize before the test: replace names with Customer A/B, order numbers with fake ones, strip addresses and phone numbers. Never feed raw tickets with real customer PII into a third-party agent platform’s eval run without checking its data-retention terms.
How often should I re-benchmark?
Monthly, and after every change to the knowledge base, tools, or model. Track the resolution rate and cost per resolution on a simple trend line — the direction matters more than any single number.
Methodology
This protocol is a synthesis of standard evaluation practices (fixed task sets, blind scoring, stratified sampling) adapted for solo operators who can’t afford a 500-ticket labeled dataset. The ±14-point confidence band for n=30 at 80% is the standard binomial standard error (√(0.8×0.2/30) ≈ 0.073, × 1.96 ≈ ±0.143). The escalation-miss tracking exists because, in our view, mis-escalation is the most expensive failure mode in support agents — one wrong refund costs more than a month of API spend. The ElevenLabs affiliate figure (22% recurring for 12 months on Starter/Creator/Pro/Scale; 11% on Business) was verified against 2026 affiliate-program writeups in October 2026. No live 30-ticket benchmark was run for this post — this is the procedure, not a result.
Sources
ElevenLabs affiliate program 2026 (22% recurring / 12 months): https://ai1102.vip/2026/08/10/elevenlabs-affiliate-program-2026-guide/
ElevenLabs affiliate guide, commission structure and PartnerStack terms: https://www.way2earning.com/2026/01/elevenlabs-affiliate-program/
EditorialGE, best SaaS affiliate programs September 2026 (ElevenLabs 22% for 12 months): https://editorialge.com/best-saas-affiliate-programs-for-content-creators
Acrid affiliate-engine research (DigitalOcean 10%/12mo, n8n 30%/12mo — program terms context): https://github.com/acrid-auto/acrid-brain/blob/HEAD/skills/affiliate-engine/ACTIVE-AFFILIATES.md
