LLM cost optimization is not simply a matter of choosing the cheapest model. This guide gives you a repeatable way to estimate token spend, compare model choices, reduce unnecessary context and output, and verify that savings do not damage answer quality, latency, or reliability.
Overview
The cost of an AI feature is usually driven by four variables: request volume, input tokens, output tokens, and the price assigned to each token category by the selected model or provider. A useful first estimate is:
Estimated cost = requests × ((input tokens ÷ 1,000,000) × input price + (output tokens ÷ 1,000,000) × output price)
Use the provider's current pricing units when completing the calculation. Some services distinguish between cached, uncached, batch, or real-time processing, so the formula may need additional terms.
Cost optimization belongs within model evaluation and reliability work. A lower bill is not a successful outcome if the model produces more hallucinations, misses required fields, calls the wrong tool, or creates additional review work. Establish a quality baseline before changing prompts, retrieval settings, routing rules, or models. For example, measure answer correctness, citation or grounding quality, structured-output validity, refusal behavior, tool-call accuracy, latency, and human-review rate.
For applications that use retrieval, cost can also rise when a knowledge base returns too many or poorly targeted chunks. A recurring content review can help: see the RAG knowledge base maintenance workflow for a practical way to keep retrieved context relevant.
How to estimate
Start with observed traffic whenever possible. A short production sample is more useful than a purely theoretical average because real requests often include retries, long conversations, tool calls, empty results, and unusually large inputs.
- Measure request volume. Record requests per day or month, then separate ordinary traffic from batch jobs, scheduled tasks, and peak periods.
- Measure input tokens. Include system instructions, user text, conversation history, retrieved documents, tool definitions, and serialized data. Do not count only the visible user message.
- Measure output tokens. Capture both successful responses and responses that are truncated, retried, or rejected by validation.
- Count extra model calls. An AI agent may make a planning call, one or more tool-selection calls, a retrieval call, and a final response call. Calculate each stage rather than treating the workflow as one request.
- Apply current rate cards. Enter the relevant input and output rates from the provider documentation or your contract. Keep these values in configuration rather than hard-coding them into a report.
- Add operational overhead. Include retries, fallback calls, evaluation traffic, background summarization, and safety or classification checks. If a value is uncertain, model a low, expected, and high case.
A simple worksheet can use these columns:
| Field | Example input | Why it matters |
|---|---|---|
| Monthly requests | Observed count | Sets the main volume multiplier |
| Average input tokens | Measured average | Reveals prompt and context cost |
| Average output tokens | Measured average | Reveals verbosity and completion cost |
| Retry rate | Observed percentage | Captures failed validation and transient retries |
| Fallback rate | Observed or planned | Captures calls to a second model |
| Input and output rates | Current provider values | Converts usage into spend |
Calculate baseline spend first. Then change one variable at a time in a copy of the worksheet. This makes it possible to attribute savings to shorter prompts, lower output limits, caching, routing, or a different model instead of combining changes that are difficult to evaluate.
Inputs and assumptions
Model selection
Compare models against the job they perform, not against a single general-purpose label. A compact model may be appropriate for classification, extraction, routing, or simple transformations. A more capable model may be justified for ambiguous reasoning, complex tool orchestration, or high-consequence decisions. Test both on a representative evaluation set.
| Decision factor | Questions to test |
|---|---|
| Quality | Does the model meet correctness and structured-output thresholds? |
| Context | Can it handle the required prompt, history, and retrieved material? |
| Latency | Does response time fit the user or workflow requirement? |
| Reliability | Does it follow tool schemas, formatting rules, and refusal boundaries? |
| Economics | What is the complete cost per successful task, including retries and review? |
Prompt and context assumptions
Reduce prompt tokens without removing information the model needs. Remove duplicated instructions, stale conversation turns, unused tool schemas, repeated examples, and retrieval chunks that do not support the current question. Summarize long histories when the summary has been tested for loss of critical details. In a RAG system, improve query construction and filtering before simply increasing the number of retrieved documents.
Output controls are equally important. Set a reasonable maximum output length, request the required format, and avoid asking for explanations that the product does not use. For extraction tasks, a concise schema is often easier to validate than a long natural-language response. Use the same evaluation set after each reduction so token savings are measured against quality rather than assumed.
Caching, batching, routing, and fallbacks
Cache stable material such as repeated instructions, deterministic lookups, or previously computed results where the application can tolerate the cache's freshness. Check provider-specific cache rules before estimating the benefit. Batch non-urgent work when the provider supports it and when delayed processing is acceptable.
Routing sends different tasks to different models or workflows. A router can direct simple requests to a lower-cost path and escalate ambiguous cases. Keep the router's own call in the cost model. Fallbacks should be explicit: define which errors trigger them, cap repeated attempts, and record whether the fallback actually improves task success.
For reliability-sensitive applications, connect cost changes to a prompt testing framework. The prompt regression testing workflow provides a useful model for comparing versions before release.
Worked examples
Consider an illustrative support assistant with 100,000 monthly requests. Assume each request averages 1,500 input tokens and 400 output tokens. These figures are examples only; replace them with measured usage and the current rates for your selected provider.
Monthly input tokens are:
100,000 × 1,500 = 150,000,000 input tokens
Monthly output tokens are:
100,000 × 400 = 40,000,000 output tokens
Enter those totals into the provider's current input and output rates. Next, test a prompt revision that removes 300 input tokens per request while preserving the required instructions and retrieved evidence:
100,000 × 300 = 30,000,000 input tokens avoided per month
Multiply the avoided tokens by the applicable input rate to estimate gross savings. Then compare the revised prompt with the baseline on answer correctness, groundedness, escalation rate, format validity, and latency. If output length also falls by 50 tokens, calculate that reduction separately.
Now compare routing. Suppose 70% of requests can pass a tested lightweight path and 30% require the original model. Calculate the cost of both paths, including the router call, fallback attempts, and any increase in retries. The routing strategy is worthwhile only if its total cost and operational impact are lower while quality remains within the agreed threshold.
Do not judge a model by token price alone. A cheaper response that requires manual correction can cost more than a longer response that is correct on the first attempt. Track cost per successful task alongside cost per request:
Cost per successful task = total model spend ÷ tasks meeting the quality threshold
This measure is especially useful for agents, classification pipelines, and structured extraction, where a technically completed call may still fail downstream validation.
When to recalculate
Recalculate the estimate whenever a pricing input, workload pattern, or quality assumption changes. At minimum, review it during a monthly usage review and before launching a high-volume feature. Also recalculate after:
- Changing models, providers, API tiers, or pricing plans.
- Rewriting a system prompt or adding examples, tools, or policy instructions.
- Changing conversation-history limits, summarization, or retrieval top-k settings.
- Introducing caching, batching, routing, retries, or fallback behavior.
- Adding an agent step, evaluator, classifier, or background process.
- Observing a change in traffic mix, output length, latency, or review volume.
Keep a versioned record of the inputs, formula, evaluation set, quality thresholds, and provider rates used for each estimate. When results change, label whether the cause was volume, token usage, price, model behavior, or workflow design. This turns cost optimization into an engineering feedback loop rather than a one-time cut.
For a practical next step, export a representative sample of requests, remove sensitive data, and calculate average and high-percentile input and output tokens. Build a baseline worksheet, evaluate one reduction at a time, and release only changes that preserve the quality measures your application depends on. For related reliability work, review the LLM hallucination troubleshooting workflow and the guide to system prompt design.
