In Retrieval-Augmented Generation, input tokens dominate up to 90% of the invoice. Here is how to compress context, tune chunking, and slash retrieval costs.
Retrieve top-20 chunks cheaply via vector search, then use Cohere Rerank or BGE to send only the top-3 best matches to your frontier LLM. Halving context halves your input bill.
If 80% of your user queries reference the same product docs or codebase overview, structure your system prompt with the fixed corpus first so it hits provider KV caches.
Excessive chunk overlap (e.g. 200 tokens) causes duplicate context to be billed multiple times across neighboring chunks. Reduce overlap to 30–50 tokens.
Pre-generate concise summaries or parent-document pointers instead of dumping entire 2,000-word raw text passages into the generator prompt.
Use metadata filtering: if a direct factual match is found with >95% confidence score, route the response generation to a budget model like GPT-5.6 Luna or Gemini Flash.
RAG payloads are input-heavy: every query injects 5–15 retrieved document chunks (typically 4,000 to 16,000 tokens) into the context window on every turn. Input tokens represent 75–90% of a RAG pipeline's bill.
Re-ranking allows you to retrieve a broad set of candidates (e.g. top 25 chunks) with cheap vector search, then use a tiny, inexpensive re-ranker model to compress the set down to only the top 3 most relevant chunks before sending them to the LLM.
Yes, if you order chunks consistently or prepend a static documentation index. For enterprise support bots querying the same handbook, caching the base knowledge chunks yields up to 90% savings on input tokens.
Smaller chunks (250–500 tokens) with semantic boundaries minimize extraneous irrelevant text, reducing input token waste compared to large 1,500-token chunks.