Harness choice has little effect on task success rate, but can significantly affect the cost on the benchmarks we test. The same model can achieve similar success rates at up to 5x costs.
Pi, a minimal, open-source harness can be competitive on both cost and task success rate.
Models may perform better with other harnesses than with their own.
New harness out from Unreal Labs. Quoting from their article – they are claiming better cost performance vs. Pi:
GPT-6 Astra · xhigh. Codex (lb) is the leaderboard baseline; Unreal Agent and Pi runs are linked below.
I’m using t3.codes as it mainly just wraps around the official harnesses and allows awesome functionality on top of it across all the models. In general I try to stick to the official tooling as much as possible due to the pacing of change/releases. Most of the benefits I get from skills.