「最好的 eval 工具是哪个」是个错误的问题。Hamel 让 LangSmith、Braintrust、Arize Phoenix 三家完成同一份作业并全程录像评审，从中提炼出四条不会过时的选型标准。附录的厂商快照会过时（作者自己提醒），但四条标准不会。
正在选型或准备换型的团队：先用四条标准搭自己的评估矩阵，再看附录的厂商快照（注意其为 2025 年中信息）。与 a14（自建 vs 采购）连读：先决定买不买，再决定买哪家。
先合上原文在心里作答，再点开对照。全对 = 本篇真正通关。
没有工具在所有维度都最优：最佳选择取决于团队技能、技术栈与成熟度；且工具迭代极快，逐功能对比很快就过时。
① 工作流摩擦（从发现失败到迭代修复的时间）；② 人在环路支持（标注与错误分析是否一流）；③ 透明可控 vs 黑箱「魔法」；④ 生态集成 vs 围墙花园（警惕私有 DSL，数据必须可导出）。
让 AI 生成评分 rubric，再让 AI 按这个 rubric 立即打分——两层未经人工验证的自动化叠加，缺陷被看似漂亮的高分掩盖，制造虚假的信心。
上方为本站原创导读，下方为英文原文的完整中文翻译（经分析→翻译→审校→润色全流程精翻）；英文原文折叠收于翻译之后，可展开逐句对照。
过去一年，无论咨询还是教学，我的工作都高度聚焦在 AI 评测上。被问得最多的问题是：「评测工具到底哪家最好？」我一直不肯直接回答，原因有二。第一，大家太在意工具、不在意流程，总以为买个工具就能开箱即用地解决问题——而它几乎从来不能。第二，工具迭代太快，今天写的对比，明天就过时。
主流评测工具我用过不少，可以实话实说：没有任何一家在所有维度上都领先。哪家的工具「最好」，取决于你团队的技能构成、技术栈和成熟度。
所以，与其做一张逐项打分的功能对比表，不如带你看看另一个更有价值的东西：一群懂评测的数据科学家，究竟是怎么评估这些工具的。在我的 AI 评测课程里，我们请三家最主流的厂商——LangSmith、Braintrust 和 Arize Phoenix——完成了同一份课程作业。同一道题、三家同场作答，这种观察机会非常难得。
整个过程和现场点评我们都录了下来，视频见下。希望它能帮你搞清楚：给自己的团队选工具时，到底该看些什么。
感谢 Shreya Shankar 和 Bryan Bischof 与我一起担任评审。
对谈嘉宾：Harrison Chase，LangChain CEO。
对谈嘉宾：Wayde Gilliam，Braintrust 前开发者关系负责人。
对谈嘉宾：SallyAnn DeLucia，Arize 技术 AI 产品负责人。
以下主题在评审中反复出现。
减少摩擦，比任何单个功能都重要。说具体点：你要盯的是「从发现一个失败，到迭代出修复方案」一共要花多久。举个例子，我们很看重这样一种连贯性：正在查看一条 trace，一键就能把它带进 playground 里做实验。对有数据科学背景的团队，以 notebook 为中心的工作流往往最理想——透明、可控。我自己也最偏好这种。
考虑 notebook 工作流时，还要掂量 SDK 顺不顺手——这一点通常就看两件事：文档质量，以及和现有数据工具的集成程度。
好工具不想着把人自动化掉，而是让人更高效。错误分析是 AI 工程里投入产出比最高的动作，工具能不能支撑高效的人工审查，因此至关重要。优先挑那些对手工标注和错误分析有一等支持的工具。截至本文写作时，很多工具还缺一样东西：轴心编码（axial coding）（一种质性研究方法：先开放标注、再归纳归并失败类别）。
凡是承诺「全自动、不用人看」的功能，都要深存一分怀疑——它们会制造一种强大而危险的信心幻觉。比如这类设计就要警惕：让 AI agent 先生成评估标准，紧接着又用它给输出打分。这种「抽象叠抽象」的玩法，常常把缺陷藏在漂亮的高分后面。优先选把控制权和可见性交给你的工具。
评测工具应该迁就你的技术栈，而不是逼你迁就它。先看它和现有技术集成得怎么样；警惕私有 DSL（领域特定语言），它们只会增加摩擦；最后，能把数据导出成通用格式、在各种环境里继续分析，是硬性要求。
选哪个工具，取决于你团队的工作流、技能构成和具体需求。希望评审团这次的评估方式，能给你一个更好用的决策框架。
至于我自己，我习惯把这些工具当后端数据仓库用，日常大部分需求靠 Jupyter notebook 加自己定制开发的标注界面解决。
以下笔记请带着保留态度读。建议先看上面的视频，了解我们是怎么应用这些标准的——你的需求不同，结论可能也不同。
总体印象：整体工作流直观，对刚接触正式评测流程的团队尤其友好。UI 会一步步引导你建数据集、跑实验、标注结果。
好评 / 我们喜欢的
批评与待改进
总体印象：评审团对 Braintrust 总体偏正面，认可它干净的 UI 和结构化的评测路径。对人在环路工作流的侧重是它的显著强项。
好评 / 我们喜欢的
批评与待改进
总体印象：评审团对 Phoenix 总体偏正面，一位评审称它是自己「最喜欢的开源评测工具之一」。定位是开发者优先、以 notebook 为中心的平台。
好评 / 我们喜欢的
批评与待改进
Over the past year, I’ve focused heavily on AI Evals, both in my consulting work and teaching. A question I get constantly is, “What’s the best tool for evals?”. I’ve always resisted answering directly for two reasons. First, people focus too much on tools instead of the process, thinking the tool will be an off-the-shelf solution when it rarely is. Second, the tools change so quickly that comparisons become outdated immediately.
Having used many of the popular eval tools, I can genuinely say that no single one is superior in every dimension. The “best” tool depends on your team’s skillset, technical stack, and maturity.
Instead of a feature-by-feature comparison, I think it’s more valuable to show you how a panel of data scientists skilled in evals assesses these tools. As part of my AI Evals course, we had three of the most dominant vendors—Langsmith, Braintrust, and Arize Phoenix complete the same homework assignment. This gave us a unique opportunity to see how they tackle the exact same challenge.
We recorded the entire process and live commentary, which is available below. We think this might be helpful in learning about the kinds of things you should consider when selecting a tool for your team.
Thanks to Shreya Shankar and Bryan Bischof for serving as the panelists (alongside me).
With Harrison Chase, CEO of LangChain.
With Wayde Gilliam, former developer relations at Braintrust.
With SallyAnn DeLucia, Technical AI Product Leader at Arize.
Here are themes that consistently surfaced during our review.
Reducing friction is more important than any single feature. Concretely, you should be mindful of the time it takes to go from observing a failure to iterating on a solution. For example, we appreciated the ability to go from viewing a single trace to experimenting with that same trace in a playground. For some teams with data-science backgrounds, a notebook-centric workflow is ideal as it provides transparency and control. This happens to be my preferred workflow as well.
When considering a notebook-centric workflow, its important to pay attention to the ergonmics of the sdk. This often boils down to the quality of the documentation and integration with existing data tools.
The best tools don’t try to automate away the human; they empower them. Since error analysis is the highest ROI activity in AI engineering, a tool’s ability to support efficient human review is paramount. Prioritize tools with first-class support for manual annotation and error analysis. As of this writing, one thing that is missing from many tools is axial coding.
Be deeply skeptical of features that promise full automation without human validation, as these can create a powerful and dangerous illusion of confidence. For example, be wary of features where an AI agent both creates an evaluation rubric and then immediately scores the outputs. This “stacking of abstractions” often hides flaws behind a high score. Favor tools that give you control and visibility.
An eval tool should fit your stack, not force you to fit its stack. Assess how well a tool integrates with your existing technologies. Also, beware of proprietary DSLs as they can add friction. Finally, the ability to export data into common formats for analysis in a variety of environments is a must-have.
The right choice of tool depends on your team’s workflow, skillset, and specific needs. I hope seeing how our panel approached this evaluation provides a better framework for making your own decision.
As for me personally, I tend to use these tools as a backend data store and use Jupyter notebooks as well as my own custom built annotation interfaces for most of my needs.
You should take these notes with a grain of salt. I recommend watch the videos above to get a sense of how we applied these criteria and where you might differ according to your neeeds.
Overall Sentiment The overall workflow is intuitive, especially for those new to formal evaluation processes. The UI guides you through creating datasets, running experiments, and annotating results.
Positive Feedback / What We Liked
Critiques and Areas for Improvement
Overall Sentiment The panel had a generally positive view of Braintrust, highlighting its clean UI and structured approach to evaluations. The tool’s emphasis on human-in-the-loop workflows was a significant strength.
Positive Feedback / What We Liked
Critiques and Areas for Improvement
Overall Sentiment The panel had a generally positive view of Phoenix, with one panelist calling it one of his “favorite open source eval tools.” The tool is positioned as a developer-first, notebook-centric platform.
Positive Feedback / What We Liked
Critiques and Areas for Improvement