Research · Updated 24 Oct, 02:51 pm IST
Study finds fancy agent harnesses raise cost but not reliable performance gains
Why it matters for readers: It explains why different evaluation setups can give very different-looking results even with the same models.
- Previous reported gaps of more than 20 points between harnesses often come from single runs and are inflated by run-to-run noise.1
- Pure reruns of the same setup varied by up to six points on SWE-bench Verified, showing substantial measurement noise.1
- The team ran five models (Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, HY4-Preview) through three shipped harnesses: Claude Code, OpenCode, and mini-SWE-agent.1
- mini-SWE-agent provides only a Bash tool and skips context management while Claude Code and OpenCode include specialized file and search tools plus history compression and longer system prompts.1
- With web access enabled early on, 71% of Deepseek and Qwen attempts retrieved fixes from GitHub, so the researchers discarded those runs and completed tests offline.1
Get a brief like this every morning
Uzha reads hundreds of sources and gives you the stories that matter for your work, with every source linked. Free.
Get started