ServiceNow 与 McGill 发布 AgentHorizon 基准：LLM 判官难察长程计算机使用任务的细微失误
ServiceNow 与 McGill 发布 AgentHorizon 基准，发现 LLM 判官难察长程计算机使用任务的细微失误
ServiceNow 与 McGill 发布 AgentHorizon 基准，包含 1,373 个来自人类屏幕录制、指令互换的近似任务判题，用于检验 LLM 判官能否识别长程计算机使用任务中的细微失误。最佳判官 GPT-5.5 在 Codex 中于困难子集取得 80.9% 平衡准确率（随机猜为 50%），Qwen 3.6 27B 直接读完整运行得 77.6%，改为智能体形式反而降至 70.4%。
New ServiceNow and McGill paper finds that AI judges still miss subtle mistakes in long computer-use tasks, and giving them tools does not reliably help.
Teams use LLMs to grade agent runs for training and evaluation. A run of 100+ steps can look finished while it drew the wrong shape or posted to the wrong Slack channel.
They built AgentHorizon, 1,373 judging tasks from human screen recordings. People recorded pairs of near-identical tasks and swapped the instructions, so each failure is a clean run of a slightly different request.
The best judge, GPT-5.5 in Codex, scored 80.9% balanced accuracy on the hard split, where always saying yes gets 50%. Qwen 3.6 27B did worse with tools, scoring 77.6% reading the whole run directly but 70.4% as an agent.
Before you trust an LLM judge with agent runs, test it on near-miss failures inside the exact agent setup you will use.
来源：Rohan Paul · x.com