How context filtering cuts token costs while improving relevance

AI cost control starts with useful computation

Minimizing tokens, the units of text an AI model processes and generates, can make an application cheaper and less useful at the same time. An extra relevance check before generation may help a production system, yet forcing its answer into too few tokens may reduce accuracy. The useful question is whether each unit of computation serves the task. Five architectural decisions apply that test: model selection, answer reuse, context reuse, context selection, and output control.

To apply that test, teams need to connect spending to application behavior. Over the last decade, FinOps, the practice of managing technology spending through financial and engineering accountability, helped organizations understand cloud bills and eliminate waste such as idle compute. Generative AI makes this harder because automated agents and prompt templates can hide which behavior consumes tokens and what business value it creates. A hypothetical feature engine spending thousands of dollars a month on pleasantries or low-value information processing has an architectural problem, even if its large language model (LLM) has a competitive token price.

These problems often begin with reasonable prototyping shortcuts: choose the strongest model, supply oversized inputs, pass retrieved material through without filtering, and impose an output ceiling when responses become verbose. Such choices help teams “get it working,” but production requires examining their consequences separately. Tokens need the same resource discipline as compute and storage, while application programming interface (API) spending needs to remain traceable to the work that caused it.

Match model capability to the task and make spending attributable

The strongest-model shortcut is a useful place to start because the model that removes uncertainty during prototyping can become an expensive production default. A highly capable model such as Claude Opus makes it easier to explore requirements without repeatedly running into capability limits. Once the task is understood, however, the engineering question changes: what is the least powerful model that performs it adequately? Simple classification and parsing workloads should be tested against their specific requirements.

For a small application, that evaluation can begin with manual tests of different models on its tasks. Larger systems can use conditional routing, which selects a model according to the request type. Frameworks such as RouteLLM and Semantic Router can direct simple classification, basic text parsing, and user-intent detection to lower-cost utility models such as GPT-4o mini or Claude 3 Haiku. A simple request then avoids the cost of the application’s most demanding use case.

Routing is easier to manage across services when it runs at a shared infrastructure layer. An AI API gateway, such as Kong, Cloudflare AI Gateway, or Portkey, can centralize model selection and spending allocation. It can also implement “cascade routing”: when the primary model encounters rate limits or a latency spike, the request falls back to a cheaper or open-source alternative. Model choice becomes an operational policy that responds to service conditions.

The same gateway can tie consumption to the tenant or microservice responsible for it. A tenant is a customer or organization sharing an application; a microservice is an independently operated component of that application. Consolidated usage records let teams enforce hard token budgets at either level. A cheaper model lowers unit cost, while attribution reveals which parts of the application consume the budget and gives teams a way to constrain them.

Those controls add infrastructure expense and complexity, so their value depends on the workloads they govern. Commercial incentives also matter: model providers earn revenue from paid usage, and commercial gateway suppliers benefit when teams adopt their infrastructure. A separate deployment choice is “repatriation of compute,” in which teams run models, often open-source models, on local hardware instead of rented cloud endpoints. Local execution brings the operating responsibilities of other on-premises systems, so both deployment and gateway decisions require a comparison of total operating cost.

Reuse answers when the underlying intent repeats

Once a request reaches an appropriate model, the next question is whether it needs fresh generation at all. Conventional caching can use Redis to associate an exact query with a stored response, but language variation limits exact matching. “How do I reset my password?” and “I forgot my login info” illustrate differently worded requests that may express sufficiently similar intent. Whether they should receive the same answer depends on how the application handles those requests.

Semantic caching extends reuse by matching meaning. An embedding model converts the incoming prompt into a numerical representation suitable for similarity comparisons, and a search compares it with previously answered prompts. If a match meets a chosen confidence threshold, 0.92 is one illustrative setting, the application returns the stored answer. It avoids calling the generative model for that request.

A qualifying hit makes the generative inference cost exactly zero and can move response latency from seconds to milliseconds. The lookup still requires work: the application must tokenize the prompt, or divide it into model-readable units, call an embedding model such as text-embedding-3-small, and perform a vector search to find nearby numerical representations. GPTCache offers a purpose-built implementation, while pgvector can store and compare embeddings within an existing PostgreSQL database. These operations earn their place when they avoid enough generation to cover their own cost.

The matching threshold determines how much nuance reuse preserves. An overly strict threshold rejects useful matches, while an overly loose threshold returns generic responses to questions that deserve different treatment. Repetitive retrieval-augmented generation (RAG) workloads, which retrieve external information to help a model answer, can benefit alongside support bots and internal knowledge bases when, for example, users ask the same 20 questions a thousand different ways. Open-ended creative work offers less reliable reuse, and a high aggregate hit rate can hide poor treatment of nuanced requests.

Reuse stable context separately from cached answers

A genuinely new question may still rely on background material the model has already processed. Prompt caching retains that contextual input within the AI engine so later questions can reuse it. The material might come from a RAG pipeline, a database, or another contextual data source. The model still generates an answer to each incoming question, so prompt caching saves a different kind of work from returning a stored answer.

Because generation continues, the savings come from reducing repeated processing of contextual input. Cached contextual tokens can receive discounts of 50% to 90% relative to ordinary input tokens, with the precise treatment varying by provider. Reuse can also reduce latency, but the new question and generated answer remain part of the workload. Providers benefit commercially from paid inference and cache-related usage, so the relevant comparison is the application’s full bill under its actual reuse pattern.

OpenAI’s automatic prompt-caching mechanism shows how request layout affects reuse. It has a 1,024-token minimum threshold and requires an exact, byte-for-byte prefix match beginning at token zero. A timestamp or user ID at the beginning of system instructions changes the prefix and prevents reuse of the material that follows. Reusable instructions and context therefore need a stable leading position, with changing values outside that prefix.

Explicit caching adds a choice about retention through approaches associated with Anthropic Claude and Google Gemini. Depending on the approach, an application can upload static context through a dedicated endpoint and assign a time to live, the period it remains available, or insert cache-control breakpoints into a JSON request payload, a structured text format used by APIs. This can involve an initial “write premium,” while allowing substantial contextual material to remain available for hours rather than minutes. The decision depends on how often the application will reuse that material during the paid retention period.

Filter context before generation to improve cost and relevance

Even inexpensive context reuse leaves a more basic question: which information belongs in the prompt? A context window is the maximum input and output a model can handle together, and capacities of one million to two million tokens make very large inputs possible. Sending an entire PDF, an entire codebase, or the last three years of logs delegates relevance selection to the generation model. The application pays to process that material before it knows how much will help answer the question.

The selection problem remains even after the application chooses a well-matched model. Model selection determines which system performs the work; context selection determines what work it receives. Oversized input can also cause the problem described as the “muddy middle,” or “context rot,” in which the model loses resolution over information within the middle of a large payload. Cost and quality can therefore deteriorate together as irrelevant material makes important information harder to use.

RAG narrows the available material by retrieving information relevant to the question and supplying it as context. A vector database supports retrieval by finding text whose numerical representation resembles the query’s representation, but similarity alone does not establish that every returned passage helps answer the request. A basic implementation may concatenate the top 20 retrieved results directly into the prompt because this connects retrieval to generation with little additional logic. The generation model then processes every passage, including those too weakly related to justify inclusion.

Reranking adds a relevance check between those stages. A small cross-encoder, a model that evaluates a query and candidate passage together, scores how well each passage addresses the user’s actual request. Cohere Rerank and open-source BGE models are implementation examples; Cohere has a commercial interest in paid use of its reranking service. This additional model makes a focused selection before the larger, more expensive generation model begins its work.

Consider a separate illustrative workflow that retrieves 50 potential matches from a vector database. The broad candidate set preserves the chance of finding useful material, after which the application supplies the candidates and the user’s question to the cross-encoder. It then keeps the top three or four highly relevant chunks for the generation prompt. Retrieval supplies possible evidence, and reranking selects the passages that most directly support the requested answer.

Narrowing the selection changes the generation model’s job because more of its input directly concerns the question. With unfiltered context, it must first find useful evidence inside a larger payload and then construct an answer from it. Filtering moves the first decision into a dedicated step, but its value depends on retaining the information needed for a correct answer. The illustrative counts show how the stages fit together; the useful configuration depends on the task and the retrieved material.

The dedicated step has its own cost: a cross-encoder may add 100 milliseconds of latency, while prompt-token reductions can reach 80% or more alongside improved final-answer accuracy. These are potential trade-offs rather than fixed outcomes for every application. Teams must compare the added delay and processing expense with the contextual work avoided and the resulting answer quality. A small relevance model can eliminate much more expensive processing, but a smaller prompt is a poor result if filtering removes evidence the answer needs.

These workload-specific effects also limit how teams should forecast savings. Cache hits, cached context, model routing, and reranking change different parts of the processing path, so their separate savings claims cannot be added into a universal forecast. A production comparison must account for which requests use each mechanism and what infrastructure, retention, and processing charges remain. The same full-cost test applies to controls on the answer itself.

Remove output filler while preserving useful reasoning

Output control matters because generated tokens commonly carry a 3x to 5x price premium over input tokens, although the relationship varies by model and provider. In an API response, “Certainly! Here is the JSON you requested,” and “Please let me know if you need anything else!” add cost without helping the receiving software use the requested data. Across repeated application calls, these additions become worth controlling. The aim is to remove unnecessary conversational text while preserving a complete, usable response.

A generation ceiling handles a different risk: runaway output. The max_tokens setting should be a safeguard because reaching it can end generation before the required structure is complete. If the response is JSON, cutting it off midway may make it impossible for the application to parse. A lower token count can therefore leave the call without a usable result.

Stop sequences offer a more targeted way to end generation after the required work. For a response containing only SQL, the language used to query databases, a marker such as Explanation: can stop an additional explanatory section; an end-of-code-block sequence can do the same when the response format uses a Markdown code block. The term stop_words is also used for this type of control, though API interfaces differ. The application must choose a boundary that preserves the complete query, including avoiding a code-block delimiter that would stop generation at the opening fence.

For machine-readable responses, structured outputs can enforce a defined schema, a set of rules for the response’s fields and data types, while strict JSON mode can constrain the response to valid JSON. An explicit system instruction can reinforce the required behavior: “Output ONLY valid JSON. Omit all conversational text and pleasantries. Failure to do so will break the system”. Here JSON is the response format consumed by downstream software, whereas the earlier cache-control example used it to carry a request. Format controls and clear instructions address the requirement directly, reducing reliance on an artificially short generation limit.

Reasoning-heavy tasks limit how far teams can suppress output because models can use generated tokens for intermediate computation, often called chain-of-thought reasoning. Forcing a complex logic task immediately into a single boolean true or false can sharply reduce accuracy by leaving insufficient room to “think” through the task. Preserving that computational capacity does not require exposing intermediate reasoning to users. It requires output constraints that leave the model enough capacity to reach a reliable answer.

Key executive takeaways

• Match models to tasks: Engineering teams can test lower-cost models against specific workload requirements and route requests accordingly. Centralized usage tracking ties spending to tenants or services and supports enforceable token budgets.
• Reuse answers for repeated requests: Semantic caching avoids fresh generation when requests express sufficiently similar intent. Application owners need similarity thresholds that preserve meaningful differences between questions and savings that cover embedding and lookup costs.
• Cache stable context: Platform teams can reduce repeated input processing by placing reusable material at the start of prompts and keeping changing values outside that prefix. Compare provider discounts, retention charges, and actual reuse before committing to explicit caching.
• Filter context before generation: Retrieval teams can use reranking to select the passages most relevant to each question, reducing prompt size and sharpening the evidence available to the model. Validate that savings and answer quality justify the added processing and latency.
• Preserve complete, reliable outputs: Application developers can use schemas, JSON constraints, and targeted stop sequences to remove filler while preserving usable responses. Treat token ceilings as safeguards and test reasoning-heavy tasks for accuracy under output constraints.