Move evaluation closer to consequences
Research on locating hidden failures in long-horizon agents supports an important methodological point: a benchmark can show that an agent reaches a goal while missing the damage created along the way. Embedded evaluations observe the actual workflow, including tool use, recovery, and the need for human intervention.
Define a failure inventory
List failures that a normal success metric hides: wrong recipient, stale context, duplicated action, unreported uncertainty, and unsafe retry. Then build small tests for the highest-impact cases. This is more useful than a broad claim that an agent is reliable or unreliable.
Publish the negative cases
Teams should preserve examples where a model looked polished but failed operationally. Those examples improve test design and prevent a launch narrative from becoming the only evidence the organization remembers.