The Lab is Automated Wealth Lab’s empirical testing ground. In this benchmark experiment, we set out to answer a burning question for modern business operators: Can an autonomous AI agent truly resolve tiered customer support inquiries without human intervention—and what does it actually cost?
The Experimental Setup
- Dataset: 1,000 anonymized e-commerce and SaaS support tickets covering shipping questions, refund requests, billing errors, and bug reports.
- Frameworks Tested: LangGraph, CrewAI, AutoGen, OpenAI Assistants API, and custom n8n AI agent workflows.
- Evaluation Metrics: Resolution accuracy, hallucination rate, tool-calling precision, and API token expenditure.
Key Empirical Findings
1. Simple Deterministic Workflows Outperform Autonomous Loops
Free-wheeling autonomous agents with unbounded tool-calling loops frequently got stuck in cyclic execution or made hallucinated assumptions when order data was ambiguous. Structured, deterministic pipelines with fallback checkpoints achieved a 94.2% resolution accuracy compared to only 71.8% for fully autonomous loops.
Token Costs & Economics
Average cost per resolved ticket across models:
- GPT-4o / Claude 3.5 Sonnet: $0.048 per ticket
- GPT-4o-mini: $0.0034 per ticket (14x cheaper with 92% comparable accuracy on classification)
The Lab Conclusion
Do not deploy open-ended autonomous agents directly in front of paying customers. Instead, use small, specialized models for routing and classification, and reserve reasoning models for drafting responses that pass through a human verification gate.
