Compare Models by Cost per Accepted Answer

01 / The useful price is attached to an accepted answer

A low token price does not guarantee a low cost per useful result. An inexpensive response that needs three retries can cost more than a stronger first attempt. The number to compare is cost per accepted answer: total spending across every attempt divided by the number of tasks that produce an acceptable result.

Here is a receipt-style comparison using entirely hypothetical rates and evaluation results. These are arithmetic examples, not published model prices or benchmark claims. Replace every input with your own billing records and test outcomes.

02 / Define acceptance before running 100 tasks

Choose 100 representative tasks and write the acceptance rules first. For a structured extraction job, acceptance might require valid JSON, every required field, and agreement with a checked reference. For code, it might require passing tests without manual edits.

Run the same tasks through two candidate routes. Keep the retry policy identical: at most one retry per task. Count a task as accepted only once, even if multiple attempts pass. Include failed attempts in spending. Otherwise, the comparison rewards routes for generating more responses rather than solving more tasks.

03 / Route A receipt: $0.627 for 84 accepted tasks

Suppose Route A makes 100 initial attempts and 20 retries. Across those 120 attempts, its billing record shows the following mutually exclusive token categories. Each rate is hypothetical and quoted per million tokens.

• ■ Uncached input: 240,000 tokens at $1.00 = $0.240.
• ■ Cache reads: 720,000 tokens at $0.10 = $0.072.
• ■ Cache writes: 60,000 tokens at $1.25 = $0.075.
• ■ Output: 60,000 tokens at $4.00 = $0.240.
• ■ Total API spending: $0.627.

If 84 distinct tasks pass the acceptance rules, cost per accepted answer is $0.627 divided by 84, or approximately $0.00746. That is 0.746 cents per accepted answer. The remaining 16 tasks are unresolved, not free successes.

04 / Route B receipt: $1.162 for 94 accepted tasks

Suppose Route B needs 100 initial attempts and 10 retries. It accepts more tasks, but its hypothetical token rates are higher.

• ■ Uncached input: 220,000 tokens at $2.00 = $0.440.
• ■ Cache reads: 660,000 tokens at $0.20 = $0.132.
• ■ Cache writes: 60,000 tokens at $2.50 = $0.150.
• ■ Output: 55,000 tokens at $8.00 = $0.440.
• ■ Total API spending: $1.162.

With 94 accepted tasks, Route B costs approximately $0.01236 per accepted answer, or 1.236 cents. Route A wins on this metric despite its lower acceptance rate. Route B solves ten additional tasks for $0.535 more API spending, but whether that trade is worthwhile depends on what happens to failures.

05 / Add unresolved work as a separate receipt line

Suppose each unresolved task requires exactly $1 of manual work to finish. This is another explicit assumption, not a labor-cost benchmark. Route A then costs $16.627 to finish all 100 tasks: $0.627 in API charges plus $16 in manual work. Route B costs $7.162: $1.162 plus $6.

The all-in cost per finished task becomes roughly $0.1663 for A and $0.0716 for B. Keep this measure separate from API cost per accepted answer. Both are useful; they answer different purchasing questions.

06 / Build the comparison from actual billing categories

Record task ID, attempt count, uncached input, cache reads, cache writes, output, billed cost, and acceptance. Use the billing system's category definitions so cache writes are not accidentally charged twice in your spreadsheet. Sum underlying charges before rounding the final metric.

CheapRelay can provide a pay-as-you-go setting for this comparison: billing is per real token, including cache reads and writes, without a monthly fee or subscription. Use your actual applicable rates rather than the illustrative prices here. Sign up for CheapRelay, run a small fixed evaluation set, and compare receipts against accepted tasks—not just advertised token prices.