We changed requirements in a coding benchmark and found models talking themselves out of following them.Sometimes the agent understood the new requirement perfectly. It described what needed to change, considered an implementation, and then decided the hidden tests probably wanted something else. In one run, a model implemented the change correctly, asked another model for the benchmark’s original answer, and removed its own correct code after receiving it.This is an awkward habit for a coding a...