How we test, and why you can trust the result

We sell evidence. Evidence is only worth something if nobody can buy a better result, and if the method is fixed before anyone sees the numbers. These are the rules we follow on every test, public or private.

1. The plan is fixed before the first call

• Before any model is called, we write down the question, the tasks, the configurations, how each answer is graded, the stop rules and the cost cap. The plan gets a date and a version hash.
• After the first call we only fix crashes that do not change the logic, and we log each fix. A new idea becomes a new test, never a change to a running one.
• For public results we publish the plan next to the result. For private audits the client receives it before we start.

2. The agent never sees the answers

• Expected answers and hidden tests stay on our side. Where it matters, they are sealed by a hash before the run, so they cannot be edited after the results are known.
• Grading is deterministic where possible (exact match, numbers with a tolerance, structured fields, unit tests). Where a judgement is needed, we say how it was made.

3. We name what you lose

• Every report lists the tasks each cheaper setup gets wrong, not only the average. Critical cases are agreed and fixed before measuring; we never add cases afterwards to change a verdict.
• Small samples get ranges (we print the interval), not certainties. Public benchmarks are labelled as such: the comparison is fair, the absolute rates are not a forecast.
• Costs are tokens × the provider's list price unless stated otherwise, and we say when they are not yet reconciled with the provider's invoice.

4. Who pays, and what money cannot buy

• No vendor pays to appear in, or to improve, our public results. We take no advertising, sponsorship of a ranking, or affiliate fees from model providers, routers or tool vendors.
• A vendor may commission a test. If we publish it, the report names who paid, on the first page. The vendor sees the plan before the run and cannot change the result.
• Client audits are private. We publish a client case only with the client's written consent.
• Anything that could look like a conflict of interest is listed in section 7, with the date we disclosed it.

5. Right of reply and corrections

• Any vendor or project named in a public result can send us a factual objection with evidence. We answer within 5 working days and publish the outcome next to the result, and the vendor may attach a short comment.
• Published results are never silently edited. A correction is a new, dated note that says what changed and why.

6. Data

• Client data is used only for that client's engagement, in an environment separate from every other client, and deleted within 14 days of the report (confirmed in writing). See our data processing terms and terms.
• We run tests through the providers' APIs under their business terms (no training on API data by default). Raw records we publish contain no client data and no personal data.

7. Disclosures

• We are customers of the model providers we test (we pay for API usage at list prices). We receive no discounts, credits or payments that depend on results.
• If we join a provider's partner programme, we will list it here with the date. Membership will not change how we test or what we publish.
• Other projects of the same team are separate from this service and never use client data.

8. How we count savings when you pay from savings

Added 12 November 2026. If you choose to pay a share of verified savings instead of a fixed audit fee, this is how we count them, in plain terms.

• Baseline: your own bills for an agreed period before the change.
• Normalised for volume: we compare cost per task or per request, so more or less traffic, or a different mix of task types, is not counted as savings.
• Provider price cuts do not count. Both periods are priced at the same list prices, so a provider's own price change is not counted as our saving.
• New costs are subtracted, for example the extra retries, longer prompts or extra checks a cheaper setup needs.
• Duration and cap: 12 months, with a cap agreed up front. Our share is 20–25% of the net saving. No verified savings means no savings share.
• The final terms (percentage, cap, baseline period and how savings are measured) are agreed in writing before the work starts, as our terms say in section 3.

Questions or objections: hello@agenttrustapi.com. This policy follows the spirit of the AMTSO testing standard used by independent security test labs (plan before test, disclosure of who paid, right of reply).