$20 to $6.50: Stop Paying for the Same Answer

01 / The receipt: $20 without a response cache

The cheapest API call is the one your application does not send. If users repeatedly ask the same stable question, paying to generate the answer every time is avoidable spending. Store a successful answer on your side, then return it when an equivalent request arrives.

Here is an illustrative monthly workload, not a benchmark or a CheapRelay price quote. Assume 10,000 requests, each using 1,000 input tokens and 250 output tokens. For easy arithmetic, assume input costs $1 per million tokens and output costs $4 per million tokens. Replace those rates with your selected model’s current prices.

• ■ Input: 10,000 × 1,000 = 10 million tokens; cost: $10.
• ■ Output: 10,000 × 250 = 2.5 million tokens; cost: $10.
• ■ Total without application-side caching: $20.
• ■ Average generation cost per request: $0.002.

02 / A 70% hit rate removes $14 of generation charges

Now assume 7,000 requests can reuse valid stored answers. Only the remaining 3,000 requests reach the API. That 70% hit rate is a scenario assumption, not a promised result.

• ■ Input for 3,000 misses: 3 million tokens; cost: $3.
• ■ Output for 3,000 misses: 750,000 tokens; cost: $3.
• ■ API subtotal: $6.
• ■ Illustrative monthly cache infrastructure allocation: $0.50.
• ■ Total with caching: $6.50.
• ■ Net saving against $20: $13.50, or 67.5%.

The $0.50 allocation is a planning assumption, not a hosting quote. Include storage, lookups, network charges and any extra compute in your own receipt. Engineering time is also a cost; this example excludes it. At a small request volume, building an elaborate cache can cost more than it saves.

03 / Your answer cache is not a provider’s prompt cache

Provider-side prompt caching can reduce the cost of processing repeated input. It still involves an API request, and output generation can remain billable. An application-side response cache returns the completed answer without contacting the model for that hit.

CheapRelay bills per real token, including cache reads and writes, with no monthly fee or subscription. Those upstream cache charges are distinct from your own response storage. Our receipt assumes ordinary input and output billing on misses, with no provider-cache adjustment. If your actual usage includes billed cache reads or writes, include those line items when calculating the cost avoided.

04 / Put every answer-changing setting in the key

A cache key based only on the visible question is usually too weak. Two identical questions can require different answers when the instructions, model, source documents or user permissions differ.

• ■ Include the exact model identifier and relevant generation settings.
• ■ Include system instructions, conversation context and the requested output format.
• ■ Include document versions, tool results or other answer-changing inputs.
• ■ Scope entries by tenant, user or permission set when answers contain private information.
• ■ Add a cache-schema version so application changes can invalidate old entries.

Start with exact-match caching. Normalize harmless formatting only when it cannot change meaning. Similarity-based matching introduces another judgment call: a nearby question is not necessarily the same question.

05 / Expire answers before stale information becomes expensive

Good candidates include repeated explanations of versioned documentation and transformations of unchanged text. Live balances, current inventory and personalized account details need stricter freshness rules or no response caching at all.

Set expiration according to the underlying data, not an arbitrary universal duration. Invalidate entries when their sources change. Store only successful, validated responses, and coalesce simultaneous identical misses so ten arriving requests do not trigger ten generations. Apply access controls and retention limits to stored answers.

06 / Measure avoided dollars, not just cache hits

Record hits, misses, cache overhead and estimated generation cost avoided. A hit replacing a long answer saves more than one replacing a short answer. In this example, the $0.50 infrastructure allocation breaks even at 250 hits, before engineering costs.

CheapRelay’s pay-as-you-go model makes that comparison straightforward: fewer upstream requests mean fewer billable tokens, with no monthly subscription to offset the saving. Sign up for CheapRelay, then measure a narrow exact-match cache against your actual traffic before expanding it.