When requirements shift, coding AIs recover from just 15.4% of failures in two tries

A preprint introducing the FeatureLoop benchmark tested 12 frontier models on 61 human-reviewed tasks from 34 repositories, with requirements changing mid-project. The best model finished 72.1% of tasks, but only 15.4% of failed stages recovered within two rounds of corrective feedback. The authors also saw incomplete repairs and regressions.

Photo: 花浅葱 / Wikimedia Commons, CC0 (cropped)

Sources

• Research arXiv（査読前） (2026-10-26)
  FeatureLoop: Benchmarking Coding Agents under Evolving Requirements

Editorial policy ・ Terms of use