When you hand customer support to an AI agent, “it seems to work” isn’t a measurement. Subydoo runs against a custom eval system: golden-answer test cases, a flex-pattern scorer, and a penalty model that treats hallucination as worse than silence. The suite started at 77 cases and has grown since.
The golden-answer dataset
This didn’t start when I built the AI. It started at launch. From day one, the website logged every support ticket that came through the contact form, and hundreds accumulated over the months. On the Discord side, nearly two years of support history across 700+ tickets sat in thread archives.
By the time I built the eval there was a large corpus to work from. The raw data went through several passes: deduplication to strip repeated questions, filtering to surface the best human responses, then a manual review where I rewrote each answer line by line into the most acurate version I could. The target was not what the human actually said, which was sometimes terse or incomplete, but what the ideal response should have been. Those became the golden answers.
The first version of the dataset was 77 cases across six groups. Every case carries the original customer message, account metadata, the raw human response for reference, and the hand-written golden answer used for scoring.
{ "id": "REAL-14", "group": "cross_domain_routing", "ticket": { "subject": "Can't download my files", "body": "I paid for commercial but..." }, "expected_handling": "direct", "must_include": ["entitlement", "download link"], "must_not_include": ["refund", "cancel"]}The original six groups were billing knowledge, cross-domain routing, regression tests, data quality, website and tech issues, and Discord transcript scenarios. 54 of those cases expect the bot to handle the ticket directly and 23 expect it to escalate to a human. Newer groups cover design assistance for mockups and the general questions N3D members ask most often.
The scoring algorithm
The scorer doesn’t look for exact strings. It uses a flex-pattern system with 120+ patterns that map each assertion to an array of regex alternatives. If the golden answer says “never expires,” the scorer also accepts “doesn’t expire,” “locked in,” “no rush,” and a dozen other phrasings.
// Weighted scoring: silence > hallucination > wrong routingconst SCORES = { must_include_hit: +1, // correct fact present must_include_miss: -1, // correct fact missing must_not_include_hit: -2, // hallucination penalty (2x) escalation_correct: +2, // right routing escalation_wrong: -3, // wrong routing (safety-critical)};The penalty hierarchy is deliberate. Missing a fact costs 1 point, stating something wrong costs 2, and getting the escalation decision wrong costs 3. That mirrors the real-world consequences: saying nothing is better than saying something wrong, and routing a billing dispute to auto-send is worse than both.
How it runs
The runner executes every case sequentialy and resets the agent session before each one for clean isolation. Every ticket is prefixed with an [EVAL MODE] tag that tells the bot to skip all side effects (no Discord messages, no draft queue, no database writes) and output only the admin review card as plain text.
Each case gets a 240-second timeout. Results save incrementaly after every case, so a crash at case 50 doesn’t lose the first 49. The runner supports resume, filtering by case ID, and a dry-run mode for testing the harness itself.
Model shootout
With the eval in place I ran five models head to head to pick the production model. The four that mattered:
MiniMax M2.7 won. GPT-4.1 Mini was disqualified despite a decent score because it failed two escalations on safety-critical cases: a dispute threat routed to auto-send, and a bug report that should have gone to dev. The suite treats routing errors as safety issues rather than quality issues, and that is what disqualified it.
Production has since moved to MiniMax M3 for tool-calling work, with Kimi models for code and chat. The eval is how those decisions get made.
The philosophy
Three principles drive the design. Golden answers are ideals, not echoes: the human reference is kept for context, but the golden answer is what the bot should say given the domain knowledge, not what a person happened to reply. Assertions test concepts, not wording: the flex patterns let the bot paraphrase freely while still holding it to the facts. And the eval is a safety gate as much as a quality metric: routing errors are scored like the incidents they would cause, not like missed facts.