Pick a Unit Test Model With a $1 Trial Budget

Start with a $1 trial budget, one small function, and your existing test runner. Your goal is to find a model that produces useful tests at an acceptable total cost. A cheap response becomes expensive if you spend half an hour fixing assertions that misunderstand your code.

The $1 figure is a suggested experiment budget, not a quoted model price or a promise about how many tests it buys. Actual spending depends on the current token rates, the context you send, and the output you request.

01 / Choose one small function before choosing a model

Pick code whose intended behavior you can explain without asking the model. Avoid starting with a database service, authentication flow, or module that needs six external dependencies. Those introduce setup problems that make a first comparison harder to judge.

For example, use a shipping_fee function that accepts an integer subtotal in cents. Its contract says shipping costs 500 cents below a 5,000-cent subtotal, becomes free at 5,000 cents, and rejects negative subtotals with ValueError.

That gives you concrete checks: zero, 4,999, 5,000, 5,001, and a negative input. Decide those expectations yourself before generating tests. Otherwise, a confident answer can quietly become your specification.

02 / Give two candidates the same small assignment

You could shortlist DeepSeek V4 Flash and Claude Haiku 4.5. Treat them as candidates to evaluate on your repository; their names alone do not establish which will cost less or produce better tests. Check their current rates before sending requests.

CheapRelay provides access through one API key, with OpenAI-compatible routes under /v1 at https://api.cheapapirelay.com. Its pay-as-you-go billing has no monthly fee or subscription, which fits an occasional testing experiment. Confirm the available model identifiers before configuring your client.

Give each candidate the same function, its contract, your test framework, and one existing test showing local conventions. Leave out unrelated files. Smaller relevant context reduces input tokens and makes the assignment easier to assess.

A suitable prompt is: “Write pytest unit tests for shipping_fee using the supplied contract and implementation. Cover ordinary inputs, the free-shipping boundary, and negative inputs. Use the existing test style. Do not change production code or add dependencies. Return only the test file.”

03 / Split the $1 budget before making requests

Reserve up to $0.20 for each candidate's initial generation and keep $0.60 for repairs or another representative function. These are spending allocations, not estimates of either model's actual charges. Check expected request costs first; reduce the context or skip a candidate if it cannot fit.

For a request containing 6,000 input tokens and 2,000 output tokens, multiply each count by its corresponding rate. If rates are listed per million tokens, the calculation is 0.006 times the input rate plus 0.002 times the output rate. Include separately billed categories where applicable.

CheapRelay bills real token usage, including cache reads and writes. Check usage after each attempt and count repair requests toward the same trial budget. Set a modest output limit, but leave enough room for a complete test file.

04 / Check whether the generated tests catch a mistake

Run each generated file with your normal test command. Review failures before changing anything: the model may have invented an import, misunderstood the contract, or exposed a real implementation bug.

• ■ Check that assertions match the written contract.
• ■ Confirm that tests cover both sides of the threshold and the threshold itself.
• ■ Look for duplicate cases and unnecessary mocks.
• ■ Record API spending and the minutes you spend repairing the result.

Then temporarily change the free-shipping condition from “at least 5,000” to “greater than 5,000.” The boundary test should fail. Restore the implementation afterward. This simple check shows whether the tests detect that specific mistake; it does not prove complete coverage.

05 / Keep the model that needs less paid cleanup

Choose using total API cost, repair time, and correct behavior coverage. If both candidates perform similarly, prefer the cheaper result. If neither handles the contract reliably, improve the supplied context before funding more retries.

Sign up for CheapRelay when you are ready to run the trial, then use the same small comparison whenever your testing workload changes.