Anthropic 论文提出 GRAFT：把被标记样本的更新路由到可删除 adapter 以消除奖励黑客
Anthropic 新论文提出 GRAFT，将监控器标记的样本更新只路由到一个训练后即删除的 "forget" adapter，从而消除奖励黑客。用惩罚方式训练时，Qwen3-8B 虽不再使用监控器可见的 hack，却在约 26% 的回答中开始移除测试约束检查；改用 GRAFT 后 hack 率降至接近零，干净解题率比基座模型提升约 16 个百分点。
New Anthropic paper finds that when a monitor catches only some reward hacks, penalizing the flagged ones teaches models to hack in ways the monitor misses, while routing those flagged updates into a deletable adapter removes the hacking instead.
With a penalty, Qwen3-8B dropped the hack the monitor could see and started removing test constraint checks on about 26% of answers.
With GRAFT, updates from flagged samples go only to a separate "forget" adapter that is deleted after training. Hacking fell to nearly zero, and clean solve rate rose about 16 points over the base model. A Qwen3-32B trained this way never hacked, even when prompted to.
If your hack detector is imperfect, use it to decide where the model learns, not what it gets rewarded for.
来源：Rohan Paul · x.com