A $10 Test Budget for Judging AI Output

You do not need an evaluation platform, a benchmark subscription, or a paid dashboard to find out whether an AI model actually does your job well. Everything below runs on a plain text file, a spreadsheet, a timer, and one cheap API key. The total outlay for the month I worked through this method was 10 US dollars, and about four hours of my own time. The process is deliberately boring, because boring processes survive contact with real invoices and real deadlines.

01 / Cap the whole exercise at 10 dollars and log every token

Decide your ceiling before you touch a single model. Mine was 10 dollars, split like this: 6 dollars for the main runs, 2 dollars for repeat runs, and 2 dollars as slack for mistakes, retries, and prompts that were broken on the first try. If you skip the cap, you will end up rerunning pretty prompts all afternoon and calling it research.

Log spend as you go rather than at the end. Billing on a pay-as-you-go relay counts real tokens, and that includes cache writes and cache reads, so an input token you assumed was free may not be. If you ignore cached tokens, your spreadsheet will look better than your invoice, and the whole point of this exercise is that the two match.

Give every row of the log these fields:

• ■ date and time, so you can rerun the same state later
• ■ model name and the exact prompt identifier
• ■ input tokens, cache write tokens, cache read tokens, and output tokens recorded separately
• ■ cost for that call, in dollars and cents to four decimal places
• ■ your pass or fail mark and a one-line reason

On CheapRelay this is simple to set up, because one key reaches every model on the platform and there is no monthly fee to amortize. A wasted call costs you the tokens and nothing else. For the bulk of my cheap runs I used Claude Haiku 4.5, and I spent the extra cents on one heavier model only for the prompts I could not get right.

02 / Build 40 real prompts from your own past work

Skip the public benchmark sets. They measure someone else's job. Open your last month of actual tasks, pick 30 of them, and turn each one into a prompt that you have already answered yourself by hand. You now have a reference answer that is worth more than any leaderboard, because you know what good looks like for your specific work.

Add 10 more prompts to probe the edges: 5 that are genuinely hard, and 5 where you deliberately leave out a piece of information the answer needs. Those last 5 tell you whether the model asks a clarifying question or quietly invents the missing number. Hallucination on a missing detail is the most expensive failure mode most beginners never test for.

Each prompt needs a short acceptance card in the same file:

• ■ the facts that must appear in the answer
• ■ the required format, such as a table, a list, or a fixed number of paragraphs
• ■ a hard length limit in words
• ■ one or two things that must not appear, like made-up citations or filler phrases

Forty prompts is enough to spot a pattern and small enough that you can score every answer by hand in an evening. Keep the file in plain text so you can diff it against next month's version.

03 / Score each answer with a 1 to 5 rubric you can repeat

Scoring is where beginners drift into liking and disliking. Fight that with a fixed rubric and four axes: correctness, completeness, instruction following, and tone or format. Score each axis from 1 to 5, write one short sentence per axis, and set your acceptance rule before you see any results. Mine was simple: an answer passes only if every axis scores 4 or higher. One axis at 3 makes the whole answer a fail.

Hide the model name while you score. Print the prompt and the answer with no header, shuffle the stack, and score in that order. It removes the part of your brain that wants the expensive model to win because it is expensive. Also timebox yourself: 90 seconds per answer, then move on. If you spend five minutes per answer, 40 answers become three hours and you will quit the method.

Two more rules keep the scores honest:

• ■ Score the answer against the acceptance card, never against your memory of what the prompt was
• ■ If a colleague is available, have both of you score the same 10 answers; if you differ by more than 1 point on 3 or more answers, rewrite the rubric wording before continuing
• ■ Record the time each review took, since a correct answer that takes you 2 minutes to verify is not the same value as one you can skim in 20 seconds

By the end of this pass you have 40 rows of numbers instead of 40 feelings. That is the entire product of the exercise.

04 / Turn tokens into a cost per accepted answer

Raw quality scores are only half the story. Divide total spend by the number of answers that passed your rubric and you get the number that matters for a tiny budget: cost per accepted answer. Worked example, using placeholder rates so you can swap in your own: if your runs used 250,000 tokens in total and your blended rate were 0.50 dollars per million tokens, the month costs 0.125 dollars. If 22 of your 40 answers passed, one accepted answer costs about 0.0057 dollars, or roughly six hundredths of a cent.

Do the same math per model, and then per prompt category. Often the cheap model handles the routine 30 prompts at a fraction of the cost while failing only on the 5 hard ones. The rational move is then obvious and much cheaper than a blanket upgrade: keep the cheap model for routine work and route only the hard category to the stronger model. Because billing is per real token with no subscription, that routing decision costs you nothing to make and can be reversed tomorrow.

Watch your cache columns while you do this. Long system prompts reused across many calls generate cache writes and cache reads, and both are billed. If the cost per accepted answer climbs as you add examples to the prompt, your prompt is getting longer faster than your pass rate is improving, and that is a trade you want to see on a spreadsheet rather than feel at the end of the month.

05 / Run each prompt three times and keep a dated baseline

One run per prompt is a coin flip dressed up as data. Run each of the 40 prompts three times, with the same settings each time, and record all three scores. A prompt that passes twice and fails once is not a pass. Mark it unstable, and treat unstable prompts as the place to rewrite the instructions before you blame the model.

Three runs triple your token use, which is exactly why the budget cap comes first. In my case three runs of 40 short prompts stayed well inside 10 dollars, because the prompts were the kind of thing you actually send at work rather than 20-page documents.

• ■ Store the raw three-run scores, not just the average, so you can see spread
• ■ Flag any prompt where the answers contradict each other on facts
• ■ Note the prompt rewrite you tried and the score change it produced
• ■ Date every baseline so next month you compare deltas instead of vibes

Rerun the same 40 prompts once a month. If the mean score moves from 3.6 to 4.1 after you rewrite six prompts, that is a real, defensible win you can put in front of a manager, complete with a cost per accepted answer before and after. If you want to run this loop yourself, sign up at CheapRelay and work with a single key across models, paying only for the tokens you actually send.

The final receipt is small on purpose: 40 prompts, 3 runs each, a rubric you can explain in 60 seconds, a spreadsheet with token columns that reconcile to your invoice, and a monthly spend you set before you started. That is enough to make better model and prompt decisions than most teams make with tools costing a hundred times more.