Spend $0 First on Your Embedding Pipeline

01 / Start with a $0 baseline

A retrieval pipeline can spend money before a user asks a question: embedding every document, re-embedding unchanged files, and sending long retrieved passages to a language model. Keep the first version small. Use a handful of representative documents, a local database or a free development setup, and a fixed set of questions. The goal is to find out whether retrieval helps before paying to scale it.

Write down three counts: documents indexed, chunks created, and queries tested. Also record whether each answer found the right source. These measurements are not a benchmark; they are a baseline for your own material. Without them, it is easy to mistake a bigger bill for better search.

02 / Reduce the text before embedding it

Clean the input before chunking. Remove repeated navigation, boilerplate, duplicate pages, and content your users would never search. Split by meaningful boundaries such as headings or paragraphs, then set a modest maximum chunk length. Overlapping every chunk adds repeated tokens and storage, so begin with no overlap or a small overlap only where a sentence might otherwise be split.

Keep useful metadata, such as document title, section, and update time, alongside each chunk. Metadata can help filter results without embedding every label into the text. If a file has not changed, reuse its existing vector. A content hash is a simple way to detect changes and avoid paying to embed identical text again.

03 / Try cheaper retrieval before buying more vectors

For a small collection, test ordinary keyword search first. It can work well when users know names, product codes, or phrases from the source. If results miss paraphrases, combine keyword search with vector search rather than replacing everything at once. A small candidate set is cheaper to inspect than ranking every chunk for every query.

Limit retrieval to a few relevant chunks, and set a maximum context budget for the answer model. More retrieved text is not automatically more useful: irrelevant passages consume input tokens and can bury the evidence that matters. Log the query, selected chunk IDs, and token counts, while avoiding sensitive user data in logs.

04 / Measure spend per useful answer

Estimate costs from actual token counts and the current rates for the services you use. Keep embedding, storage, retrieval, and answer-generation costs separate. The total for a test run is the sum of those components; there is no universal price per document because text length, model, and usage differ. Include cache reads and writes when your provider bills for them.

Compare a few versions on the same questions: keyword-only, vector-only, and a hybrid if needed. Note which sources were retrieved correctly and whether the final answer used them. If a change raises cost but does not improve your test results, remove it. Recheck after changing chunk size, document filters, or the model.

05 / Keep paid usage predictable

Put limits around indexing and queries: cap batch size, skip unchanged content, restrict retrieved context, and alert when usage passes a budget you choose. Separate experiments from production credentials where possible. Track spend by task so one large re-index does not look like ordinary daily traffic.

If you need a language model for answers, use a model suited to the task and compare its real token usage on your own prompts. CheapRelay offers pay-as-you-go access through one API key, with no monthly fee or subscription; its OpenAI-compatible routes and Gemini-native route can fit common client setups. Pricing still depends on the selected model and tokens consumed. Sign up for CheapRelay when you are ready to test paid model calls, and start with a small, capped workload.

A tiny budget is a reason to measure earlier, not to skip retrieval design. Begin with a $0 baseline, avoid embedding unchanged or irrelevant text, and pay only when a measured improvement justifies the added cost.