自动化 Evals 真的有效吗？——六大系统 vs 人工标注的实测
关于「自动化错误分析」目前唯一的对照实验：100 条真实生产 trace、领域专家标注出 39 个故障，Braintrust Loop、Arize Alyx、LangSmith、Codex、Claude Code 等 6 个系统同场竞技。结论用数据说话：最佳召回 87%，每个系统还额外发现 17-20 个人工遗漏的问题，但四类「看似正确」的故障全部漏网。想用这类工具，先读这篇。
核心内容速览
- 可复用的实验设计：专家标注 → 掩码 → 统一 prompt → 按 Recall / Precision / Discoveries / 误报四个指标打分，任何团队都能照此评估自己的自动化工具。
- 六系统成绩单：Braintrust Loop 召回最高（87.2%）但误报偏多；Arize Alyx 精确率最高（91%）；编程 Agent（Codex / Claude Code）与专用平台在发现质量上基本打平——平台的优势在分析之外的工作流（发现与 trace 关联、复核标注、持续重跑）。
- 自动化补人的盲区：每个系统都发现了 17-20 个人工评审遗漏的次要问题——自动化不是替代人，而是与人互补。
- 共同的系统性盲区：销售异议处理、SMS 里的 Markdown 格式、语音打断、漏转人工——这些故障「trace 看似正确但没达成真实目标」，需要 trace 之外的业务上下文（criteria drift），没有工具能开箱发现。
- 推荐姿势：不做一次性错误分析；人留在环路中持续标注，AI 从标注中实时学习故障分类法，并带回新疑似病例由人确认——每轮把更多产品上下文注入评测体系。
适合谁 · 怎么用
已经在用或正考虑引入 Loop / Alyx / LangSmith Engine 等自动化分析工具的团队必读：先看清楚它们能抓到 87%、漏掉哪 13%，再决定人在环路里的位置。与 a04（Grader 校准）、a09（一致率）连读。
闯关自测
先合上原文在心里作答，再点开对照。全对 = 本篇真正通关。
Q1 自动化系统漏掉的故障有什么共性？
都是「trace 看似正确、但没达成产品真实目标」的故障：销售异议、渠道格式（SMS 里的 Markdown）、打断用户、漏转人工。它们需要 trace 之外的业务上下文——属于 criteria drift，没有工具能开箱即发现。
Q2 实验里编程 Agent 和专用平台谁更强？
单看发现质量基本打平（召回 79-85%）。专用平台的优势在分析之外：发现与源 trace 保持关联、提供复核与标注的工作流、能随新 trace 持续重跑。
Q3 作者推荐的错误分析姿势是什么？
不做一次性任务：人留在环路中持续标注，让 AI 从标注中实时学习故障分类法，并把新疑似病例带回给人接受或驳回——每一轮都把更多产品上下文注入评测体系。
以下为全文 · 中文翻译
上方为本站原创导读，下方为英文原文的完整中文翻译（经分析→翻译→审校→润色全流程精翻）；英文原文折叠收于翻译之后，可展开逐句对照。
过去三年，我们回答关于 AI 评测的大多数问题，用的都是同一条建议：看你的数据。1 在实践中，「看数据」就是错误分析2——读你的生产 trace（或日志），把产品的失败方式编目归档。而读 trace 还会反过来改变你要找的东西：你需要标准来评判输出，但评判输出恰恰是你发现标准的过程——这个现象叫标准漂移（criteria drift）。正因如此，我们曾主张不要把评测完全外包给 LLM。3
而如今流行的评测工具，正在把人往另一个方向引。Braintrust、Arize、LangSmith 各自都上线了 agent（分别叫 Loop、Alyx、Engine）：读你的生产 trace，报告哪里在出问题。4 编程 agent 的能力也已经强到可以对着一个装满日志或 trace 的文件夹尝试同样的工作。我们赞成在任何有帮助的环节使用 AI，但这些工具的诱惑在于：把人完全移出环路。这就引出一个重要问题——这些系统靠自己能抓住多少？又会漏掉什么？
为了找到答案，我们做了实测。先准备数据集：来自一个生产环境公寓租赁 AI 的 100 条 trace，与一位领域专家合作，人工标注其中的失败。然后隐去这些标注，让每个系统自行审查 trace、找出错误。这样，人工与自动化错误分析就能正面对比。
自动化系统的表现令人印象深刻：最好的系统找回了人工标注失败的 87%；而且每个系统都发现了人工漏掉的问题。但缺点同样明显——全自动方案总是抓不到那些「看起来正确」、实则不够好的交互；系统还会标记出不少并非失败的问题，带来噪音。
本文余下部分讲三件事：实验怎么做、结果怎么样，以及如果由我们来设计，会怎么做。
实验设计
这些 trace 来自一个真实运作的公寓租赁 AI agent。5 潜在客户通过短信或电话咨询房源，agent 负责答疑、预约看房；按物业规定，某些对话必须转交人工。关键是，本实验用的是真实客户的 trace，而不是合成基准。6
准备数据集时，一位领域专家按照错误分析的标准流程，手工标注了 100 条 trace，最终形成一套失败分类，共 39 个已标注失败。
然后，我们隐去全部人工标注，把 trace 连同下面这个提示词交给每个系统：
@nurtureboss_evals.jsonl 是一个公寓租赁客服 AI（电话与消息渠道）的生产环境 trace。每条 trace 包含完整对话、全部工具调用及其输出。
分析这些 trace，并：
- 找出数据中反复出现的失败、风险与质量问题。
- 尽可能把相关 trace 归成问题簇，并给每个簇命名。
- 对每个簇或问题，列出示例 trace 的标识符（如元数据中的
trace_id或session.id）。 - 注明哪些 trace 看起来可以接受或无法定论——不要预设所有 trace 都有问题。
- 如果可以，提出候选评估器（评什么、怎么评），不要引用任何外部评分标准。
要具体：能引用对话轮次或工具步骤就引用。避免空泛的主题归纳，除非你能把它对应到 trace 里的证据。
这个提示词刻意从简：只简单说明产品和整理发现的方式，对我们已知的失败只字不提。根据我们教评测课程的经验，这也正是大多数人开箱使用这些产品的方式。这里有一条数据集中的匿名化 trace 示例。
另外，我们本想顺带体验各产品的工作流——期待工具在分析前追问几句、向我们澄清需求。但它们基本都没有。
我们用三个指标给每个系统计分：
- 召回率（Recall）：39 个人工标注的失败，它找回了多少。
- 精确率（Precision）：它标记的错误里有多少是真失败。标记与人工标签一致，或虽被人工漏标但确属真实问题，都计为有效。
- 新发现（Discoveries）：人工审查中没有、但有效的新标记。
看数字之前先泼一点冷水：这个实验只有一个数据集、39 个已标注失败，系统之间的差距往往就落在几条 trace 上。它证明不了哪家厂商更强——排名也从来不是目的。7 我们关心的是：这些系统各擅长什么，又共同栽在哪里。
工具们抓到了什么
| 系统 | 召回率 | 精确率 | 新发现 | 误报 |
|---|---|---|---|---|
| Braintrust Loop | 87.2% | 79.1% | 20 | 9 |
| Arize AX Alyx | 74.4% | 91.0% | 19 | 3 |
| LangSmith（对话 agent） | 79.5% | 77.5% | 20 | 9 |
| Codex（GPT-5.5 High） | 84.6% | 82.8% | 20 | 11 |
| Factory Droid（GPT-5.5 High） | 84.6% | 83.3% | 17 | 10 |
| Claude Code（Claude Opus 4.8） | 79.5% | 77.4% | 17 | 14 |
只要失败在 trace 里显而易见，每个系统都表现得很好。它们全都抓到了：没有依据的看房时间承诺、反复索要用户已经给过的确认、答应了却没有兑现的后续跟进、与工具输出自相矛盾的回答。
每个系统还额外找到了 17 到 20 个人工审查漏掉的小问题。比如有一条 trace：agent 成功预约了看房，人工审查员把它标为正常——但 Braintrust 和 Arize 都注意到，助手曾索要一条用户其实已经给过的短信确认。
Braintrust Loop
Braintrust 在三个平台里召回率最高，用起来也最顺：上传 trace、跑自动评估、导出结果，一路无卡顿。它也不会强行给每条 trace 安一个罪名——会把一些对话标为「可接受」或「无法定论」，这让它的发现更可信。
高召回的代价是误报：9 条标记是错的。通常是 Loop 对「完整性」过于严苛，或者把内部工具元数据当成了「本该告知用户」的信息。这些发现听上去都很有道理——直到我们回头重读完整 trace。
Arize AX Alyx
Arize 拿召回换精确。Alyx 漏掉的人工标注失败比 Loop 多，但 91% 的标记有效，只有 3 条是错的。8 它尤其擅长抓「没有工具输出支撑的说法」：把普通一居室叫成「loft」、把最便宜的市场价房源说成保障房选项、在没有楼层信息的情况下回答楼层问题、承诺回电却没有任何对应动作。
LangSmith
论单条 trace 的阅读界面，LangSmith 是几家里最好的。分析本身我们试了它的两个工具：Engine 正是为这个场景设计的——能发现反复出现的问题，并把每个问题链回支撑它的 trace；但它找出的问题太少，远少于完整人工审查的数量，没法用于这次 100 条 trace 的对比。正式计分实验我们改用 LangSmith 的对话 agent 来跑。
LangSmith 对话 agent 标记到真失败时，通常会引用确切的消息或工具调用，这让它的发现最容易核实。短板是严重度评级：好几条错误发现被标成高严重度——虚高的严重度会误导团队安排修复顺序。
编程 Agent
Claude Code、Codex、Factory Droid 这些通用编程 agent，和专业平台相比并不落下风。它们拿到同样的提示词、同样的遮蔽 trace。它们会引用确切的工具参数、日期、价格和互相矛盾的数值作为失败证据，还抓到了平台漏掉的微妙失败——比如把「实地看房链接」错当成「线上看房」的确认。
单看发现的质量，编程 agent 和专业平台大致打平。平台的优势在分析之外的工作流：发现与源 trace 保持关联、有地方审查和标注、还能随新 trace 流入反复重跑分析。
所有系统一起漏掉了什么
漏掉的东西并非随机。所有系统漏掉的都是同一类失败：trace 看起来没问题，但 agent 没有达到产品的真实目标。人工审查抓到了几类没有任何系统能稳定发现的问题：
- 销售异议：agent 的职责是租出公寓，但它在客户提出第一个异议时就放弃了，而不是回应对方的顾虑。
- 短信里的 Markdown：agent 用 Markdown 排版短信，到了用户手机上就是一堆乱糟糟的星号和井号。
- 打断对方：语音 agent 抢话，没让对方说完。
- 漏掉转人工：物业规则要求某些对话必须转人工，agent 却自己继续聊下去了。
这些失败只看 trace 是看不见的。一段「助手礼貌地接受了客户异议、结束对话」的记录，读起来像成功。trace 里也没有任何信息表明对话发生在短信渠道，或哪些情形按业务规定必须升级。
一个显而易见的反驳是：我们故意没给这些上下文，真实团队会给的。但团队没法提供自己尚未发现的上下文。比如我们做这个项目时，正是在读到一条「客户转身离开」的 trace 之后，才意识到需要处理销售异议。这种「看到之前，你不知道什么是好什么是坏」的现象，就叫标准漂移（criteria drift）。9
换成我们，会怎么做
我们能不能引导这些系统抓到漏掉的东西？几乎可以肯定。如果事先描述过这个产品「好」和「坏」分别长什么样，大多数漏掉的失败都在每个工具的射程之内。但刚开始做评测时，那份完美的描述几乎从不存在。
我们测过的所有系统，都没做那件本来最有帮助的事：采访我们。每个工具都把错误分析当成一次性任务——trace 进去，报告出来。
我们不会把错误分析当一次性任务来跑。我们会自己标注 trace，让 AI 从标注中持续学习。我们的同事 Shreya Shankar10 在下面的视频里演示了这种工作流：她让编程 agent 在她的 trace 之上搭一个小型审查界面，然后随手标注让她不舒服的地方。agent 实时观察她的标注（她用了 Claude 的 monitor 工具），从中学习失败分类，再把新的疑似实例带回来让她确认或排除。每一轮，都有更多她的产品上下文被拉进评测套件。做判断的始终是她，但 agent 让判断变得高效。
作为 AI 产品的建造者，你应该留在环路里，而不是把评测完全外包给 AI。如果某个工具能自己发现并修复所有问题，它也能替你的竞争对手做同样的事——那你的产品就没什么差异化了。想脱颖而出，就得亲眼看你的数据。
附：想要手把手的评测实战指导？可以了解我们的 AI Evals 课程——直播小班，带指导练习和答疑时间，链接里有读者优惠。
脚注
Hamel 与 Shreya Shankar 共同讲授 AI Evals for Engineers & PMs 课程，并合著了 O'Reilly 出版的 Evals for AI Engineers。
关于错误分析包含什么、为什么它最优先，见我们的错误分析 FAQ。
Hamel 在 The Revenge of the Data Scientist 一文中阐述了这一主张。
该公司是 Nurture Boss，一个面向公寓管理的 AI 助手。Hamel 在 A Field Guide to Rapidly Improving AI Products 中描述过他们的错误分析流程。
我们获得了该公司分析并撰文讨论这些 trace 的许可，但未获再分发许可，因此无法公开数据集。
我们与本文提到的任何厂商均无关联，也未接受其赞助。
工作流中也有磕绊：Alyx 最初只读每条 trace 的前 200 个字符，后续运行还遇到上下文压缩和超时问题。我们反馈后，Arize 团队都很快解决了。Alyx 也无法按我们要求的格式把分析写回数据集，我们只好把结果复制到电子表格里人工核对。
「标准漂移」由 Shreya Shankar、J.D. Zamfirescu-Pereira、Björn Hartmann、Aditya G. Parameswaran、Ian Arawjo 在 《Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences》中提出。
Shreya 与 Hamel 共同讲授 AI Evals 课程。她的写作与研究见 sh-reya.com。
查看英文原文（English original）
For the past three years, we have answered most questions about AI evals with the same advice: look at your data.1 In practice, looking at your data means error analysis2, where you read your production traces (or logs) and catalog how your product fails. Reviewing traces also changes what you look for. You need criteria to grade outputs, but grading outputs is how you discover your criteria, a phenomenon known as criteria drift. For this reason, we argued against outsourcing evals to an LLM.3
Popular eval tools now nudge people the other way. Braintrust, Arize, and LangSmith have each shipped an agent (Loop, Alyx, and Engine) that reads your production traces and reports what is failing.4 Coding agents have become capable enough to attempt the same job from a folder of log files or traces. We believe in using AI in any workflow where it helps. However, the temptation with these tools is to take the human out of the loop entirely. This raises an important question: how much can these systems catch on their own, and what do they miss?
To find out, we tested it. First, we prepared a dataset of 100 traces from a production apartment-leasing AI, working with a domain expert to label failures by hand. Then we masked those labels and asked each system to review the traces and find the errors on its own. This allowed us to compare human versus automated error analysis.
The automated systems performed admirably. The best recovered 87 percent of failures flagged by humans. Additionally, every system found issues humans missed. But there were downsides. Fully automated approaches always failed to catch interactions that “looked correct” but fell short of providing a great user experience. The systems also flagged a fair number of issues that were not failures, which added some noise.
The rest of this post covers the experiment, the results, and how we would modify the approach to be effective.
The experiment
The traces come from a real AI agent for apartment leasing.5 Prospects text or call about apartments, and the agent answers questions and books tours. Certain conversations are supposed to be handed to a human. Importantly, this experiment is run on traces from real customers instead of a synthetic benchmark.6
To prepare the dataset, a domain expert labeled 100 traces by hand, following the error analysis process. That review produced a failure taxonomy with 39 labeled failures.
We then masked all human annotations and fed the traces to each system with the following prompt:
@nurtureboss_evals.jsonl are production traces from an AI that handles apartment-leasing customer support (phone and messaging). Each trace includes the full conversation and any tool calls with their outputs.
Analyze the traces and:
- Surface recurring failures, risks, and quality problems you observe in the data.
- Group related traces into issue clusters where possible (name each cluster).
- For each cluster or issue, list example trace identifiers if available (e.g.
trace_idorsession.idin metadata). - Note whether any traces appear acceptable or inconclusive, do not assume all traces failed.
- If you can, suggest candidate evaluators (what to measure and how), without referencing any external rubric.
Be specific: cite conversation turns or tool steps when possible. Avoid generic themes unless you tie them to evidence in the traces.
The prompt is deliberately light. It briefly describes the product and how to organize findings, but it says nothing about the failures we already knew about. Based on our experience teaching evals, this is also how we expect most people to use these products out of the box. Here is an anonymized example of a trace from the dataset.
Lastly, we wanted to experience each product’s workflow, and we hoped the tools would ask follow-up questions or elicit requirements from us before starting the analysis. They largely did not.
We scored each system on three outcomes:
- Recall: how many of the 39 human-labeled failures it found.
- Precision: how many of the errors flagged were real failures. A flag counts as valid if it matches a human label or if it is a real issue the human review had missed.
- Discoveries: valid flags that were absent from the human review.
A caveat before the numbers. This experiment is one dataset containing 39 labeled failures, so the differences between systems come down to a handful of traces. They cannot establish that one vendor is better than another, and ranking vendors was not the goal.7 Instead, we investigate where these systems shine and where they fall short.
What the tools caught
| System | Recall | Precision | Discoveries | False positives |
|---|---|---|---|---|
| Braintrust Loop | 87.2% | 79.1% | 20 | 9 |
| Arize AX Alyx | 74.4% | 91.0% | 19 | 3 |
| LangSmith (chat agent) | 79.5% | 77.5% | 20 | 9 |
| Codex (GPT-5.5 High) | 84.6% | 82.8% | 20 | 11 |
| Factory Droid (GPT-5.5 High) | 84.6% | 83.3% | 17 | 10 |
| Claude Code (Claude Opus 4.8) | 79.5% | 77.4% | 17 | 14 |
Every system did well when a failure was obvious by looking at the trace. All of them caught unsupported scheduling claims, repeated requests for a confirmation the user had already given, promised follow-ups that didn’t happen, and answers that contradicted tool output.
Each system also found 17 to 20 minor issues that our human review had missed. For example, in one trace, the agent booked a tour successfully and the human reviewer marked the trace as fine. Braintrust and Arize both noticed that the assistant had asked for a text-message confirmation the user had already given.
Braintrust Loop
Braintrust had the highest recall of the three platforms and was easy to use. We could upload our traces, perform the auto-evaluation, and export its findings seamlessly. It also did not force an issue onto every trace. It marked some conversations as acceptable or inconclusive, which made its findings easier to believe.
The tradeoff for the high recall was false positives. Nine of its flags were wrong, usually because Loop was too strict about completeness or treated internal tool metadata as something the user should have been told. These findings sounded convincing until we reread the full trace.
Arize AX Alyx
Arize traded recall for precision. Alyx missed more of the human-labeled failures than Loop, but 91 percent of its flags were valid and only three were wrong.8 It was especially good at catching claims with no grounding in tool output. It flagged the assistant for calling ordinary one-bedroom units “lofts,” for presenting the cheapest market-rate unit as an affordable-housing option, for answering a question about floor level without having floor information, and for promising a callback that no action backed up.
LangSmith
LangSmith had the best interface for reading individual traces. For the analysis itself, we tried two of its tools. Engine is built for exactly this job. It finds recurring issues and links each one to its supporting traces. But it surfaced only a small set of issues, far fewer than a full review turns up, so we could not use it for the 100-trace comparison. We ran the scored experiment with LangSmith’s chat agent instead.
When LangSmith’s chat agent flagged a real failure, it usually cited the exact message or tool call, which made its findings the easiest to verify. Its weakness was severity ratings. Several of its wrong findings were marked high severity, and inflated severity misleads a team deciding what to fix first.
Coding Agents
General-purpose coding agents like Claude Code, Codex, and Factory Droid were competitive with the platforms. They received the same prompt and the same masked traces. These coding agents quoted exact tool arguments, dates, prices, and conflicting values as evidence for failures. They also caught subtle failures the platforms missed, including an in-person tour link presented as confirmation of a virtual tour.
Judged only on the quality of their findings, the coding agents and the dedicated platforms were roughly even. The platforms’ advantage was the workflow around the analysis. They kept findings linked to the source traces and gave us a place to review and annotate them. They can also rerun the analysis as new traces arrive.
What every system missed
The misses were not random. Across every system, the same class of failure went uncaught. In each case the trace looks correct, but the agent falls short of the product’s real goal. Our human review caught several that no system flagged reliably:
- Sales objections: the agent’s job is to lease apartments, but it gave up at the first objection instead of addressing the prospect’s concern.
- Markdown in SMS: the agent formatted text messages with Markdown, which arrives on a phone as stray asterisks and pound signs.
- Interruptions: the voice agent talked over callers instead of letting them finish.
- Missed handoffs: the property’s rules required routing certain conversations to a human, but the agent kept them.
None of these failures are visible in the trace alone. A transcript where the assistant politely accepts a prospect’s objection and ends the conversation reads like a success. Nothing in the trace indicated the conversations were over SMS, or which situations the business promised to escalate.
The obvious objection is that we withheld this context on purpose, and that a real team would provide it. But a team cannot provide context it has not discovered yet. For example, when working on this product we only discovered the need for objection handling after reading a trace where a prospect walked away. The phenomenon where you don’t know what’s good or bad until you see it is called criteria drift.9
What we would do instead
Could we have steered these systems to catch what they missed? Almost certainly. If we had described what good and bad look like for this product, most of the missed failures were within reach of every tool we tested. But that perfect description rarely exists when evals are first started.
No system we tested did the one thing that would have helped most: interview us. Every tool treated error analysis as a one-shot task, with traces in and a report out.
We would not run error analysis as a one-shot task. Instead, we would annotate traces ourselves and let the AI learn from annotations continuously. Our colleague Shreya Shankar10 demonstrates this workflow in the video below. She asks a coding agent to build a small review interface over her traces, then annotates what bothers her. The agent watches her annotations in real time (she uses Claude’s monitor tool for this), learns a taxonomy of failures from them, and brings back new suspected instances for her to accept or dismiss. Each round pulls more of her product context into the eval suite. She is still the one judging, but the agent makes the judging efficient.
As an AI product builder, you should stay in the loop instead of completely outsourcing evals to AI. If a tool could find and fix every issue on its own, it would do the same for your competitors, and there would be nothing left to set your product apart. To stand out, look at your data.
P.S. Want hands-on help with Evals? Check out our AI Evals course, a live cohort with guided exercises and office hours. There’s a reader discount at the link.
Footnotes
Hamel co-teaches AI Evals for Engineers & PMs with Shreya Shankar and co-authored the O’Reilly book Evals for AI Engineers.↩︎
For what error analysis involves and why it comes first, see our error analysis FAQ.↩︎
Hamel makes this argument in The Revenge of the Data Scientist.↩︎
The company is Nurture Boss, an AI assistant for apartment management. Hamel described their error analysis process in A Field Guide to Rapidly Improving AI Products.↩︎
We have the company’s permission to analyze and write about these traces, but not to redistribute them, so we cannot publish the dataset.↩︎
We have no affiliation with, and received no sponsorship from, any vendor mentioned in this post.↩︎
The workflow had rough spots. Alyx initially read only the first 200 characters of each trace, and later runs hit compaction and timeout problems. The Arize team resolved each issue quickly once we reported it. Alyx also could not write its analysis back into the dataset in the format we requested, so we copied results into a spreadsheet for manual review.↩︎
Criteria drift was documented in Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, and Ian Arawjo, “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences”.↩︎
Shreya co-teaches the AI evals course with Hamel. Her writing and research are at sh-reya.com.↩︎
原文链接：https://parlance-labs.com/blog/posts/auto-evals/