Pick Models by Your Prompts, Then Check the Receipt

01 / A leaderboard cannot price your next 100 tasks

A benchmark score can help you shortlist models. It cannot tell you whether a model follows your invoice schema, preserves your product terminology, or produces a patch your team will accept. Your buying decision needs a second document: a receipt for work you actually need done.

The honest question is not which model wins overall. It is which model delivers an acceptable result at the lowest total cost for your workload. That includes tokens, retries, and the minutes someone spends repairing plausible mistakes.

02 / Build a test set before opening the pricing table

Collect 100 representative prompts from your own work. Remove secrets and customer identifiers. Include ordinary requests, awkward edge cases, and tasks where failure is expensive. Keep a separate development set for adjusting prompts; repeatedly tuning against your evaluation set makes the final score less credible.

Define acceptance before running either candidate. For extraction, require valid structure and correct fields. For code, require relevant tests to pass and inspect the change. For writing, specify factual accuracy, required content, and forbidden claims. Hide model identities during review where practical.

Use the same task requirements, comparable output limits, and a consistent retry policy. Record configuration differences rather than pretending every setting has an exact equivalent.

03 / The token receipt: $0.35 versus $1.05

Here is a hypothetical budgeting example, not a published benchmark or a CheapRelay price quote. Every rate, outcome, and review time below is an assumption. Replace them with current quoted rates and your measured usage before spending money.

Assume each candidate handles 100 initial requests averaging 2,000 input tokens and 500 output tokens. That gives each candidate 200,000 input tokens and 50,000 output tokens.

• ■ Candidate A assumed rates: $1 per million input tokens; $3 per million output tokens.
• ■ Candidate A input: 0.20 million × $1 = $0.20.
• ■ Candidate A output: 0.05 million × $3 = $0.15.
• ■ Candidate A initial total: $0.35.
• ■ Candidate B assumed rates: $3 per million input tokens; $9 per million output tokens.
• ■ Candidate B input: $0.60; output: $0.45; initial total: $1.05.

This example assumes zero cache reads and writes. That is a workload assumption, not a promise of free caching. If your usage includes caching, add separate receipt lines using the applicable read and write rates. Check how the service classifies those tokens so you neither omit charges nor count them twice.

04 / Add retries and the person checking the answers

Now assume Candidate A passes 80 tasks immediately. Ten more pass after one retry each, while ten remain unacceptable. Candidate B passes 96 immediately, two after one retry each, and leaves two unacceptable. Each retry uses the same assumed token counts as an initial request.

• ■ Candidate A: 110 attempts, 90 accepted results, $0.385 in API charges.
• ■ Candidate B: 102 attempts, 98 accepted results, $1.071 in API charges.
• ■ API cost per accepted result: approximately $0.00428 for A and $0.01093 for B.

On tokens alone, A wins. Now assume A requires 30 minutes of review and B requires 10 minutes. At an explicitly assumed labor cost of $30 per hour, review adds $15 for A and $5 for B.

The combined receipt becomes $15.385 for A and $6.071 for B: approximately 17.09 cents versus 6.19 cents per accepted result. B wins under these assumptions. Neither total includes completing the rejected tasks, so report the acceptance counts alongside the costs.

05 / Keep benchmark results beside your own receipt

Use benchmarks to choose candidates, then let your acceptance criteria and measured costs decide. Keep raw responses, usage records, retry reasons, and review times. Repeat the comparison when prompts, model versions, or prices change.

CheapRelay offers pay-as-you-go access through one key with no monthly subscription, making it useful for running a bounded comparison without adding another recurring bill. Billing includes real token usage, including cache reads and writes. Sign up for CheapRelay, set a small test budget, and judge the resulting receipt—not just the ranking.