Sample report: the client is invented, the tasks and numbers are from our public Aider test. Protocol and raw data.
Diet report: "Example agency", coding agent
1. Answer first
Switch the agent from Opus to Sonnet. On your 27 test tasks it solved the same 26 at 49% lower cost (US$0.30 → US$0.15 per solved task). It lost nothing.
Keep Haiku off this work. It is over 16× cheaper again, but it failed 4 tasks that Sonnet solves.
2. Cost and quality, same tasks
| Configuration | Solved | 90% range | Cost per solved task | Saving vs today |
|---|---|---|---|---|
| Today: Claude Opus 5.5 | 26/27 | 85–99% | US$0.297 | baseline |
| Claude Sonnet 5.5 | 26/27 | 85–99% | US$0.152 | −49% |
| Claude Haiku 5.5 | 22/27 | 67–91% | US$0.009 | −97%, with losses |
At 1,000 tasks a month, the Sonnet switch saves US$145 a month with no loss on these tasks.
3. What you lose with each cheaper configuration
Sonnet: nothing that today's setup solves.
Haiku fails these, all of which today's setup solves:
- 11/54Interpreter task: arithmetic and division-by-zero errors (11 of 54 checks fail).
- 3/12Graph parser: errors for malformed input (3 of 12).
- 1/7Word game: "the same correct letter twice counts as a miss" (1 of 7).
- 8/9REST API: balances between users (8 of 9).
Pattern: the main path works; edge cases and error handling break. If those matter to your clients, stay on Sonnet.
4. How we measured
- Your tasks, your expected answers, and your current prompt and settings. Only the model changed.
- Answers were checked automatically against the expected result. The agent never saw the answers.
- Cost = tokens × the provider's list price, measured per task.
- The plan (tasks, configurations, rules, cost cap) was fixed before the first call.
5. Range and scope
- Every rate carries its 90% range, printed in section 2.
- Re-run in 30 minutes after any provider model update.
- Rare but critical cases go in a separate set, agreed before measuring. Add yours in the audit.
6. Next steps
- Switch to Sonnet on 10% of traffic for a week and compare live results.
- Re-check in 3 months, or before any model you use retires (retirement calendar).
A client report has the same six parts, uses only your tasks, and stays private. Prices · How we handle your data