Same result, half the price: a model “diet” for a coding agent

Trust Engine AI · 11 November 2026 · a small test, written down before it ran

We gave the open-source coding agent Aider 27 programming exercises with hidden tests. We ran each exercise three times, changing only the model, and counted what each solved exercise cost.

• Opus → Sonnet: the same result for about half the cost.
• Sonnet → Haiku: about 16 times cheaper again, but it failed 4 more exercises.

Averages hide what you lose

A pass rate tells you how many tasks a cheaper model drops, not which ones. For each exercise that Haiku failed and the others passed, we re-ran the hidden tests on its final code:

• 11 / 54 a small language interpreter: arithmetic and division-by-zero handling;
• 8 / 9 a small REST API: balances between users;
• 3 / 12 a graph parser: errors for malformed input;
• 1 / 7 a word game: the same correct letter twice counts as a miss.

Failing tests out of all tests for that exercise.

The main path usually worked. Edge cases and error handling were what broke. Whether that is acceptable depends on your work, so a cost decision needs the list of losses, not just the average.

How it was done

• The protocol, with the question, models, stop rules and cost cap, was written and committed before the first paid call.
• Agent: Aider 0.86.2 with its defaults. Exam: the Python part of Aider's public polyglot benchmark. The agent never saw the tests.
• Each exercise got one attempt, plus one retry with the test output, as in Aider's own benchmark.
• Cost = tokens × list price, read from the agent's own output.

What this does not show

• One agent, one language, 27 of 34 exercises. Our API credits ran out mid-run; we report every exercise that finished for all three models rather than rerun selectively.
• The benchmark is public, so the models may have seen it. Comparing the models is fair; the absolute rates are not a forecast.
• The agent fetched some web pages named in the exercises, which our protocol did not allow. We report it rather than hide it.
• Costs are estimates from token counts and are not yet reconciled with the provider's invoice.

Doing this for your agent

If your agent's bill looks too high, the useful answer is not “use a cheaper model”. It is: here is a cheaper setup, here is proof that it does your tasks as well, and here is exactly what it gets wrong. We do this on 20–30 of your own tasks with known answers, and we fix your rare but critical cases before measuring.

The CSV has one row per exercise and model. The in_analysis column is 0 for the one exercise that finished for only one model.