> Public copy of the protocol frozen on 11 November 2026, before the first paid call. Internal approval notes, machine paths and internal log references are removed; nothing about the method was changed.

# DIET-001: same quality for less money? An open-source agent on a public exam

Status: **FROZEN** 2026-11-11, before the first paid call. After the first call only crash fixes that leave the
logic unchanged are allowed, each logged. Budget: about US$5–10 for the API.

## Question
For a popular open-source agent doing real work, how do quality and cost change when only the model changes?
This is a public "before / after" example of a model "diet".

## Agent and exam (public, reproducible)
- Agent: **Aider** (open-source coding agent, `aider-chat` from PyPI, latest version at setup, version recorded),
  default settings except the flags listed below. Installed in an isolated virtual environment.
- Exam: **Aider polyglot benchmark, Python part**, all 34 exercises in `python/exercises/practice`
  (github.com/Aider-AI/polyglot-benchmark, commit recorded). Each exercise has a stub file, instructions and a test
  file. **The agent never sees the test file** (the hidden answers).
- Models (the only thing that changes): `claude-haiku-5-5`, `claude-sonnet-5-5`, `claude-opus-5-5`, each with
  Aider's defaults for that model.

## Procedure per exercise and model (the protocol Aider's own benchmark uses)
1. Fresh copy of the exercise without the test file. Aider gets the stub file and the instructions
   (`.docs/instructions.md`, plus `instructions.append.md` if present) as its message.
   Flags: `--yes-always --no-git --no-auto-commits --no-check-update --analytics-disable --no-show-model-warnings`.
2. The test file is copied back and `pytest` runs (timeout 60 s). Pass = all tests pass.
3. If it fails, one second try: Aider gets the first 50 lines of the pytest output and is asked to fix the code.
   Tests run again.
- Order: exercises in alphabetical order, models interleaved (Haiku, Sonnet, Opus for each exercise).
- Tokens per call are read from Aider's output; cost = tokens × list price (Haiku US$0.10 / 0.50, Sonnet 2 / 10,
  Opus 4 / 20 per 1M input / output); the provider's own figures are reconciled later.

## Measures
Per model: pass@1, pass@2 (out of 34, with 90 % Wilson intervals), total cost, cost per exercise, **cost per
solved exercise**, median seconds. Comparison: the cheapest model whose pass@2 is within 5 points of the best, and
its saving per solved exercise.

## Stop rules
- Running cost estimate above **US$12**: stop before the next call.
- An Aider error that is ours (install, flags), not the model's: stop, fix (crash fix only), log, rerun that cell.
- No network access other than PyPI (setup), GitHub (exam) and the Anthropic API. The key is set only in the
  process environment, never printed or logged.

## Output (review date 12.11.2026)
Per-cell results (`results.jsonl`), the run log, Aider and benchmark
versions, a summary table.

## Honest limits
- One agent, one language, 34 exercises: a demonstration, not a general rule.
- Public benchmark: models may have seen these exercises in training; the comparison between models is still fair,
  the absolute pass rates are not a forecast.
- Aider's defaults per model (thinking, edit format) differ; this is a configuration comparison as a user would run it.