$0.0031 per accepted answer: rank models by the receipt

01 / What the cost per accepted answer actually divides

Most model leaderboards rank quality and ignore the bill. You pay dollars, so a fair ranking has to include them. The cost per accepted answer is one division: total spend on a fixed set of prompts, divided by the number of answers you would actually ship. Cost per accepted answer equals total spend divided by accepted answers. That is the whole formula.

The trick is that the numerator is not the cost of the best call. It is the cost of every call you made: the answer you rejected, the retry after a timeout, the third attempt that finally produced valid JSON. A cheap model that fails four times out of five can be more expensive per usable reply than a pricier model that gets it right on the first try.

Take a tiny worked example. Your prompt plus context is 1,200 input tokens and the answer is 800 output tokens. A model priced at 1 dollar per million input tokens and 3 dollars per million output tokens costs 0.0012 + 0.0024 = 0.0036 dollars per call. Ten calls cost 0.036 dollars. If six of those ten answers pass your check, the cost per accepted answer is 0.036 divided by 6, which is 0.0060 dollars, or about six tenths of a cent.

• ■ Count in the numerator: input tokens, output tokens, cache reads, cache writes, failed attempts, retries.
• ■ Count in the denominator: only the answers that pass your acceptance check.
• ■ Fix the prompt set first, so every model answers the same questions.

Acceptance must be objective before any of this means anything. Use unit tests, a schema validator, a rubric you grade yourself, or a human yes or no. A fuzzy definition of good lets you move the goalposts after the money is spent.

02 / A $10 budget and two toy price schedules

Now the numbers that make the point. These are toy prices for teaching the arithmetic, not quotes from any provider, so do not treat them as offers. Model A costs 1 dollar per million input tokens and 3 dollars per million output tokens. Model B costs 0.25 and 1 dollar per million. Both see the same 1,200 input and 800 output tokens per call.

Model A costs 0.0036 dollars per call. Model B costs 0.0003 + 0.0008 = 0.0011 dollars per call, roughly three times cheaper at the sticker level. Now add acceptance. Suppose A passes 6 of 10 prompts and B passes 3.5 of 10. Cost per accepted answer for A is 0.0036 divided by 0.60 = 0.0060 dollars. For B it is 0.0011 divided by 0.35 = 0.0031 dollars. B wins by nearly half.

On a 10 dollar budget that is 1,666 accepted answers from A and about 3,225 from B on the same prompts. The cheap model wins this round because its quality drop is smaller than its price drop.

Flip the quality and the verdict flips too. If B only passes 1.5 of 10 prompts, its cost per accepted answer is 0.0011 divided by 0.15 = 0.0073 dollars, which is worse than A at 0.0060 dollars. The expensive model is now the frugal choice. Sticker price alone told you nothing in either case.

• ■ Sticker price answers: what does one call cost.
• ■ Cost per accepted answer answers: what does one usable reply cost.
• ■ Only the second number matches the thing you actually buy.

03 / Retries, cache reads and rejects belong in the ledger

Real workloads add three costs that toy examples skip. First, retries. If a model returns malformed output one time in four and you resend, every rejected attempt is already paid for. Keep the failed calls in the numerator and the accepted answer in the denominator.

Second, caching. Agent runs and coding tools send the same context again and again, and cache reads and cache writes are still tokens. On CheapRelay, billing is per real token including cache reads and cache writes, so your ledger should add those token counts into spend rather than assuming a cache hit is free. This is where a per-call estimate built from a single prompt can badly understate what a long session costs.

Third, throughput failures. A timeout that you retry still consumed input tokens. A truncated answer that you regenerate still counts. The honest bookkeeping rule is simple: if a token was billed, it goes in the numerator.

A workable minimum ledger per prompt is short. Record the model name, prompt id, input tokens, output tokens, cache read tokens, cache write tokens, attempt count, accepted yes or no, and the cost you computed from the live price. A spreadsheet with eight columns is enough to rank ten models.

04 / Running the trial on CheapRelay with one key

To test several models without juggling accounts, CheapRelay at cheapapirelay.com gives you one key across Claude, GPT, Gemini, DeepSeek, Kimi, GLM, Qwen, Grok and others. It is pay as you go, billed per real token, with no monthly fee and no subscription, so a small trial costs only the tokens you send. The API base is https://api.cheapapirelay.com with OpenAI-compatible routes under /v1 and Gemini native routes under /v1beta, and it works with Claude Code, Codex, Gemini CLI and any OpenAI SDK.

Start with a 25-prompt set drawn from work you actually do: five easy, fifteen medium, five hard. Run each model against the same set with the same temperature and the same acceptance check. Keep prompts short at first so a full run for one model stays in the cents.

Use the sale list to pick your first candidates, then confirm the current price in the dashboard before you run anything, since prices move and the ledger should use the live figure. DeepSeek V4 Flash and Claude Haiku 4.5 are both on sale now and make sensible first entries on a tiny budget, precisely because low sticker price plus decent acceptance is what wins this metric.

After the run, compute cost per accepted answer for each candidate. Also note the acceptance rate on its own, because a model that is cheap per accepted answer but fails your hard five is not ready for production. Keep the raw rows so you can re-run the comparison later without rebuilding the set.

05 / Promoting a winner and keeping the receipt line honest

Ranking is not a one time event. Prices change, context lengths grow, and the quality of a model on your specific prompts drifts as you change how you prompt it. Re-run the 25-prompt set whenever you change prompts substantially or whenever the price list changes, and keep the old receipt lines so you can see the trend rather than a single snapshot.

Set a budget ceiling in dollars and a ceiling in cost per accepted answer before the run, not after. On a 10 dollar month, a model at 0.0031 dollars per accepted answer gives you about 3,225 usable replies, while a model at 0.0060 dollars gives about 1,666. Write those two numbers on the receipt and the decision usually makes itself.

Two habits keep the numbers trustworthy. First, record prices in the same unit, dollars per million tokens, and convert every prompt cost to dollars so you never add tokens to dollars by accident. Second, separate cache reads, cache writes and fresh input in your counts, because the mix changes between a short chat and a long agent session and the blended cost moves with it.

If you want a low-cost place to run the trial, sign up at cheapapirelay.com and start with a few cents of tokens rather than a whole budget at once. The referral reward is 20 percent of what invited users pay, which is a small but real offset once you have a ranking worth sharing. The point of the metric is not to find the cheapest model in the abstract. It is to find the cheapest model that clears your bar on your own prompts, and to have the arithmetic ready when someone asks why you chose it.