How to cut your LLM spend

If you run anything on top of an LLM, you have probably watched the bill climb faster than you expected.

For most of machine learning's history, the expensive part was training. Now it is inference: the cost of answering requests, one token at a time, every time someone uses the product. It has become a real line item for teams of every size, and the providers clearly know it. Prompt caching and half-price batch tiers exist because tokens are the cost, and everyone is trying to use fewer of them.

The awkward part is that the bill scales with success. More users mean more calls, more calls mean more tokens, and the invoice grows every month the product does well. For a team with real traffic, monthly model spend can reach the same order as an engineer's salary, and it keeps climbing from there.

It usually shows up the moment a feature leaves the prototype and meets production traffic, which is more repetitive and more multi-turn than anything you tested with. Across the industry, enterprise spend on model APIs more than doubled in a year, from $3.5 billion to $8.4 billion, even as the price of a single token kept falling (Menlo Ventures).

Most of that spending is avoidable. A large share of a typical bill pays for tokens you did not need, or a model larger than the task required, and both can be reduced without the user noticing. This guide is how we approach that at Phantm, the LLM gateway we build: where the money goes, the techniques that bring it down, and what each one did in our evaluation of 13,491 requests, where the full pipeline cut cost 47.1 percent with no measurable loss in quality.

Where the money actually goes

Two problems drive almost every oversized bill, and both are fixable. Requests get sent to models far more powerful than the task needs, and those requests carry far more tokens than the answer requires.

The token problem is sneakier than it looks, because not all tokens cost the same. Output is priced five to six times higher than input across the major providers, so a model that answers a yes-or-no question in three paragraphs is burning the most expensive tokens there are. And context repeats: in a twenty-turn conversation, a 1,500-token system prompt is sent, and billed, on all twenty turns. Wrap that inside an agent that makes a dozen model calls for a single user action, and the waste stops being a rounding error. It becomes the bill.

Put plainly, there are four places to attack the problem: the model you choose, the context you repeat, the prompt you send, and the response you allow. The rest of this guide is each one, roughly in the order it pays off.

Right-sizing the model

Start with the model, because it is the largest single lever. Most traffic is easier than the model it runs on, and the reason that matters is the gap between tiers:

List prices per million tokens, June 2026. Sources: OpenAI, Anthropic.

A token on Opus costs five times a token on Haiku; GPT-5.4 costs more than three times its mini. Sending a simple classification to a flagship model is like couriering a postcard. If you could send the easy requests to the cheap model and keep only the genuinely hard ones on the expensive one, you would cut the model bill without touching quality. That is model routing, and it is the most popular place teams start.

It is also why so many teams reach for tools like LiteLLM or OpenRouter. Those are good plumbing: one endpoint that can talk to many models. But the plumbing is the easy part. The hard part is deciding, for each request, which model can actually handle it, and on those tools that decision is usually a rule someone wrote by hand. Send anything under 500 tokens to the cheap model. It works right up until a 200-token request asks for a contract summary, the cheap model gets it wrong, and a real customer sees the mistake. A static rule cannot tell a short easy question from a short hard one, and in production that is the entire game.

Phantm replaces the rule with a classifier. Before anything is routed, a small model we call the gate reads the request and judges how hard it actually is, what kind of task it is, and how long a good answer should be. Easy, well-defined work drops to a mini or nano tier; reasoning, code, math, and planning stay on the flagship; and the small fraction a cheaper model would get wrong are caught and up-routed back to a stronger one. If a provider errors or rate-limits, the call fails over to another. In our evaluation this moved 60.8 percent of general traffic and 72.8 percent of customer-support traffic off the baseline model, and the quality held. The lesson underneath it is simple: routing is only ever as good as the thing deciding where to route, so the accuracy of that classifier, not the plumbing beneath it, is what separates real savings from a stream of support tickets.

Never paying twice for the same context

Routing decides which model answers. The next saving is making every call carry less, and the easiest place to begin is the context you send over and over.

Most production prompts open with a large, stable block, a system prompt, instructions, a few examples, that is identical across thousands of calls. Both OpenAI and Anthropic will cache that block and bill a repeat at roughly a tenth of the normal input price, a 90 percent discount on tokens you were going to send anyway. It is the lowest-risk saving there is, because the model sees exactly the same input; you are simply charged less for the part it has already seen.

There is a catch that costs teams more than they realize. Providers only cache a prefix once it crosses a size threshold, around 1,024 tokens. A 900-token system prompt sits just under that line, so it never caches, and you pay full price on every single call, indefinitely. Phantm watches for exactly this case and pads the prompt past the threshold with stable filler, so the same 900-token prompt suddenly qualifies and starts billing at the cached rate. It is a small trick that turns a permanent full-price charge into a 90 percent discount, and it is the kind of thing you only catch if you are sitting in the request path looking at every prompt. For workloads that genuinely repeat, asking the same things in slightly different words, an optional semantic cache goes a step further and returns a stored answer for a close-enough request, skipping the call entirely, guarded by a similarity threshold so it never answers something it should not. Across our customer-support evaluation the input-token cache-hit rate reached 89.8 percent on the Anthropic track and 36.3 percent on OpenAI.

Trimming the prompt and shaping the response

Caching handles the context that repeats. The rest of the prompt, and the response that comes back, still tend to be heavier than they need to be, and two more steps run on every request to fix that.

On the way in, cleanup strips dead filler and repeated framing without touching anything structural like a code block, and pruning trims old conversation turns once a thread runs long, keeping the system prompt and the most recent exchange while dropping the least relevant middle. On the way out, output shaping caps the response to the length the task calls for and nudges it toward the format you actually need, a short answer or a JSON object instead of an essay. That last step matters out of proportion to its simplicity, because output is the expensive half of the bill. The only requests left untouched are tool calls, which need room to work. In our evaluation cleanup fired on 14 to 24 percent of requests, history trimming on 22 to 45 percent, and output shaping on nearly all of them, since almost any response can stand to be a little leaner.

The catch: all of this has to happen in real time

Here is the objection that stops most teams from doing any of this inline, and it is a fair one. Every step so far, classifying the request, trimming the prompt, checking a cache, shaping the response, is work that happens before the model is even called. Do it naively and you have traded a money problem for a latency problem, and latency is its own tax. Users abandon slow answers, and an agent that pauses even half a second on each of a dozen calls becomes unusable. This is exactly why most cost tools settle for watching from the sidelines, a dashboard that reports what you spent last week, rather than changing the request as it happens.

Phantm runs the entire pipeline in the request path, in real time, and the whole thing adds under 200 milliseconds, a P50 of 181.6 and a P95 of 206. It can do that because the part that looks expensive, the classifier, is not another API call out to a model; it is a small local model that answers in a couple of milliseconds, for a fraction of a cent, and it is logged like everything else. Streaming passes straight through. From the outside the optimization is invisible: your application sends a request and gets back a normal, streamed response, only cheaper. You do not have to choose between a smaller bill and a fast product, which is the choice most teams quietly assume they are stuck with.

How much it adds up to

On their own these are modest. Together they compound, because each one works on what the last one left behind. Take a support assistant handling a million requests a month, each carrying a 1,200-token reused system prompt plus 800 tokens of fresh input and returning 500 tokens, all on GPT-5.4.

Illustrative arithmetic on June 2026 list prices. Your real figure depends on your traffic.

The baseline is two billion input tokens at $2.50 plus 500 million output tokens at $15, or $12,500. Caching the 1,200-token prefix moves it to the cached rate and cuts input cost from $5,000 to $2,300. Shaping output from 500 tokens to 350 drops output cost from $7,500 to $5,250. Routing 60 percent of the rest to a mini tier brings the total to about $4,379, a 65 percent reduction, before batching or a fine-tuned model is even in the picture. The exact number is not the point; the point is that the techniques stack, and the inputs that decide your number, your prompt length, your output length, and how much of your traffic is easy, are things you can measure. The cost calculator runs the same math on yours.

Without losing quality

All of this is worthless if the answers get worse, and the honest fear, the one that keeps teams from optimizing at all, is that a downroute or a trimmed prompt quietly degrades quality in a way nobody notices until a customer does. The only real answer is to make every change visible and to prove the quality held.

So every decision the pipeline makes is recorded with a reason code: which model handled the request and why, what was trimmed, whether a cache was hit, and why anything was blocked. You can read it per request, which means the optimization is never a black box rewriting your prompts behind your back. The only extra model call in the whole path is the gate, and because it costs a fraction of a cent and is logged like the rest, nothing happens off the books. Cost itself is held in place by policy rather than trust, through per-tenant budgets, rate limits, and an allowed list of models that fail closed, refusing a request they cannot check rather than waving it through.

The savings number is measured, not asserted. When you start, your traffic passes through untouched while the gateway measures a real baseline in parallel and compares it against the optimized path, so the figure you see is your own cost, before and after, on your own workload. We held the whole pipeline to that same standard in our evaluation, across 13,491 prompts in two phases, 6,500 on standard public benchmarks and 6,991 on customer-support workloads, against a realistic enterprise baseline.

The cost figure is only half the result. Cutting spend is trivial if you are allowed to return worse answers, so we tested quality with formal equivalence testing against a strict threshold, the same method used to show two treatments are interchangeable in clinical research, and every source landed under the line at which a person would notice a difference. The full corpus, scoring, and per-stage results are in the evaluation report and the whitepaper.

Which brings it back to that bill that rivaled a salary. The same product, the same code, the same answers, for closer to $4,400 a month than $12,500, with a quality result you can put in front of anyone who asks. That is what cutting LLM spend looks like when it happens on every request, in real time, and is measured end to end, rather than guessed at from a dashboard after the money is already gone.