In a 2024 a16z analysis, inference costs for AI-native companies reportedly account for 60-80% of total AI expenditure. I have seen the same trend in all enterprise AI projects I have been involved in the past 8 years. The prototypes are usually cheap, but when usage goes up, the invoices soon leave budget expectations. Token costs are usually to blame, and are almost invariably avoidable.
This post describes operational token optimizations for production deployments, as opposed to optimizations shown to work in a demo environment. I will explain prompt caching, context window management, model routing and output compression, as well as point out the most common areas where cost savings are most justified, and therefore most needed.
Most overrun costs in enterprise LLM deployments can be attributed to an unmanageable token volume as opposed to model selection. It is reasonable to assume integrating prompt caching and context trimming and a tiered model routing approach will lead to a 40-70% decrease in AI inference costs. While the techniques described in the post require token engineering, the return on investment will be apparent by the end of the month.
Why do token costs go up so drastically and so quickly?
Token costs sound simple to calculate, until you have to run thousands of API calls a day. Most developers build cost estimates with a simple equation. Estimated Average Tokens per call multiplied by Estimated Number of API Calls Multiplied by Per-Token Cost. This may work for a demo, but breaks in production for a few reasons.
Initially, system prompts expand as teams incorporate instructions, guardrails, format specifications, and examples. A system prompt that began as 200 tokens often ranges from 1,500 to 2,000 tokens in two months. Secondly, conversation histories build. Applications that store chat context send the entire context on each API call, and this context compounds rapidly. Thirdly, the length of the output is seldom limited. If there are no constraints, models supply more tokens than are necessary for the task.
These three factors mean that LLM cost optimization is a system design problem, and not solely a cost and pricing problem.
The Four Token Optimization Strategies Worth Building Around
1. Prompt Caching
Prompt caching is the most effective token optimization strategy for applications that have a shared system prompt or have a context block that is static. Both Anthropic's Claude API and OpenAI's GPT-4o support the caching of repeated prefix content. If a cached prefix is used again, it costs significantly less than a full input token. Anthropic's read to the cache costs 10% of a standard input token, whereas a cache write costs 125% of standard input tokens.
As an example, if your system prompt is 1,200 tokens and your application has 50,000 requests a day, most of your input tokens are spent at the cache-read price. For this to occur, your system prompt must remain stable. Any change to a cached system prompt will break the cache and will require a full-priced rewrite to the cache.
After applying this method to several enterprise clients, I found that teams who modularize their prompts, using a modular approach to isolate the static part of the system context from the dynamic part that is user-facing, found that modular prompts improved caching and resulted in input token costs that were reduced by at least 50% (estimated).
2. Context Window Management
For most applications, context window management is where the most cost is wasted. The standard behavior during a multi-turn interaction is to send the entire history with every call. This is almost always unnecessary.3. Tiered Model Routing
Not all requests need GPT-4o or Claude Opus. This is likely the most underutilized token optimization approach in the enterprise space, and it runs counter to what most teams do instinctively when building LLMs.4. Output Token Compression
While most of the focus has been directed at input tokens, output tokens tend to cost more per unit on most major providers’ price lists. Controlling the length of output is an easy lever to pull to reduce costs.- Explicit format constraints in the prompt: An output example with a structure and size specification (e.g., ‘Respond with a list of 3 items in JSON’), will usually generate a more predictable and structured output than a generic prompt.
- Max token settings in API calls: Setting max_tokens in your API call will stop the model from generating an output that exceeds a specified size. This protects you from output that is unexpectedly large and helps you estimate costs more accurately.
The second, even though it should be generally known, is not widely applied in production systems. Most teams apply it in development, and when stakeholders request longer outputs, the limit is removed and never restored.
Estimated Cost Reduction by Strategy (Bar Chart)
The following represents estimated cost reduction ranges observed across production deployments. These are estimates, not guarantees, actual savings depend on application architecture, traffic patterns, and provider pricing.
All figures are estimates based on real-world production deployments and publicly available provider pricing data.
Strategy Comparison Table
Strategy | Cost Impact (Estimate) | Implementation Complexity | Best Use Case |
|---|---|---|---|
Prompt Caching | High (45–60%) | Low–Medium | Apps with static system prompts |
Context Window Management | Medium–High (30–45%) | Medium | Multi-turn chat applications |
Tiered Model Routing | High (40–60%) | Medium–High | High-volume mixed-complexity workloads |
Output Token Compression | Low–Medium (15–30%) | Low | Any application; quick win |
All Combined | Very High (50–70%) | High | Enterprise-scale production systems |
The Silent Cost Killer: Allowing System Prompts to Expand Without Control
The title is specific because a vague title would in most cases allow teams to circumvent it. The issue is, in production, system prompt expansion is expensive, and most leads do not account for it.
Recommendation For Teams Starting Now
If you do not have a background with LLM cost management, prioritize the following instructions in the specified order.- Week one: Reduce output tokens. Set limits with API calls. Adjust output formats defined in your prompts. This requires no expenditure and results in immediate savings.
- Week two: Prompt audit. Remove instructions that do not have a measurable impact on output quality. Measure before and after, removal almost always has no impact.
- Week three: Add prompt caching. If your system prompt is (or can be made) stable by separating the static content from dynamic content, cache the prefix. For high-volume systems, this is the most impactful change.
- Month Two: Implement a tiered routing layer. To start, create a quick, inexpensive model to sort requests as simple or complex, then route requests based on this classification. Pay careful attention to the quality delta, and set routing thresholds based on what your specific case can handle.
The most invasive change to your system design will be managing the context window. For this reason, I have saved it for last. It will require a change in the way conversation states are saved and retrieved. After you have implemented context windows, this change will multiply the effect of every other optimization.
Key Takeaways: Where to Focus First
Token optimization must be a continuous process. The volume of tokens increases as users increase, system prompts increase as new features are added, and inference AI pricing will be affected as competition in the space grows. The most effective strategies for optimization in the long run are those embedded in the core of your system, as opposed to last-minute fixes.
- The first step must be output compression, as it can be implemented in an afternoon and will have an immediate positive effect.
- The system prompt must be audited on a monthly basis. The most common cause of cost growth at your organization will be prompt bloat.
- Implement prompt caching first, as the savings become more pronounced as your system scales with higher volume.
- Once you have sufficient traffic to accurately capture the value of tiered model routing, implement it. Routing without significant metrics to measure quality impacts will create unnecessary user support issues.
- Baking context window management into your core system design, as opposed to a localization retrofit, must be a part of your planning when your conversation history is growing unmanageable.
When combined, these token optimization strategies will greatly reduce the production costs for AI inference, and achieve a more sustainable budget across the organization.
Frequently Asked Questions
How can I decrease token consumption when making calls to the ChatGPT API?
There are a number of ways to achieve this. You can impose strict limits on max_tokens when making a call, you can use more concise system prompts, you can truncate previous chat history using a rolling window, and you can utilize prompt caching when dealing with static content. These methods can be used individually and will likely show savings on your account within the first billing period.
What does token optimization in LLMs refer to?
Token optimization encompasses the strategies used to decrease both the input tokens (sent to the model) and the output tokens (returned by the model) when making API calls. Cutting the number of tokens will decrease the costs of using LLMs since costs are assessed by the number of tokens used. Strategies used to optimize tokens include compression of the prompts, management of the window of context, routing of models, and constraints on output formats.
In what ways does prompt caching lower AI expenditure?
Prompt caching lowers costs by saving a static prefix (system prompt or context block) so that the system does not have to be prompted and processed again on subsequent calls. Companies like Anthropic process caching at a cost that is a small fraction of the input tokens (as low as 10% of the Claude standard). For use cases that have an extremely high volume of calls and a system prompt that rarely changes, prompt caching alone can reduce the cost of input tokens by about 40 to 60%.
What is the most cost-effective approach for using GPT-4 or other frontier LLMs at scale?
The most affordable approach is leveraging a combination of tiered model routing (e.g., routing easier tasks to smaller models like GPT-4o-mini), prompt caching to store repeated context, output token compression by using max_tokens and a structured output format, and context window trimming for use cases requiring multiple turns. Each of these techniques provides cost savings, but implementing all of them is required to save money; together, the strategies can lower the cost of AI inference usage by 50%–70% compared to the baseline cost of usage ( estimate) provided.
Takeaway
The estimates for costs provided in this article rely on the costs listed by the provider in early 2026, and those estimates will become outdated. Token costs for OpenAI, Anthropic, and other frontier model providers change often, and can change greatly, so consider any individual, per-token estimates as a guideline and not a guaranteed cost.
The cost savings listed are estimates based on the usage patterns I have observed firsthand; the savings your application will incur will be based the specifics of your application, the providers, the prompt design, and the expected user load.
James Mitchell
He is the Founder and Editor-in-Chief of Techisane. He holds a Master of Science (MS) in Computer Science and a CISSP certification, with eight years of experience in enterprise technology. He began his career working with IT infrastructure before advancing into IT security and consulting. Mitchell brings firsthand experience to his writing, drawing on technologies he has implemented, tested, and worked with in real-world environments.