Token economics: why your AI bill scales with sloppy context

Enterprises budget for AI by counting users. Models bill by counting tokens. The gap between those two curves is context — and most of it is waste you are paying to resend.

Where the tokens actually go

In a production assistant, the user's question is a rounding error. A forty-word query routinely rides on thousands of tokens of freight: the system prompt, tool definitions, retrieved document chunks, and — in multi-turn or agentic sessions — the entire accumulated history, resent with every call. RAG multiplies this: retrieval that pulls ten chunks "to be safe" pays for ten chunks on every turn, whether or not the answer needed more than one. Agents multiply it again, because each reasoning step re-transmits the growing transcript. The result is superlinear cost growth: the tenth turn of a session can cost several times the first, with no corresponding increase in value delivered. Latency degrades on the same curve, since time-to-first-token tracks input size.

Structural versus semantic compression

Not all context reduction is equal, and the distinction matters for both quality and audit. Structural compression removes what carries no meaning: boilerplate markup, repeated headers, redundant whitespace, duplicated chunks retrieved twice from overlapping windows. It is lossless with respect to meaning and safe to apply everywhere. Semantic compression makes judgement calls — summarising earlier turns, pruning chunks scored irrelevant to the current question, deduplicating near-identical passages. Done well, it preserves answer quality while cutting the payload substantially; done crudely, it deletes the sentence the answer depended on. The engineering discipline is to compress conservatively, measure answer quality continuously, and keep the aggressive modes behind explicit configuration.

Cache-aware compression

Modern inference stacks discount cached prefixes heavily — but caching rewards stable context, and naive compression destroys stability by rewriting the prefix every turn. Cache-aware compression orders the payload so the invariant parts (system prompt, tool definitions, long-lived documents) stay byte-identical at the front, and volatility is confined to the tail. Compression and caching then compound instead of cancelling: you send fewer tokens, and more of the ones you send are billed at the cached rate.

What 48–79% means for a workload

AANCER's Hermes compression layer measures 48–79% context reduction across workload types — the range reflecting how much redundancy a workload carries (chat-heavy sessions with long histories sit at the top; terse single-shot queries at the bottom). Read that as a planning number: a workload budgeted at a given monthly token spend lands at roughly a fifth to a half of it, and every downstream line item — capacity for on-prem serving, per-token fees where a cloud model is used for burst — scales with the same factor. Latency improves in parallel, which users notice before finance does.

Compression must never touch the record

One non-negotiable: compression is an optimisation of what the model sees, never of what the ledger keeps. Under EU AI Act Article 12-style record-keeping, the original prompt, the full retrieved context and the compressed payload actually sent must all remain retrievable, so an auditor can verify both what the user asked and what the model was shown. A compression layer that discards originals converts a cost saving into a compliance gap.

• Instrument tokens per interaction by component — system, tools, retrieval, history — before optimising.
• Apply structural compression universally; gate semantic compression on measured answer quality.
• Order context for cache stability so savings compound.
• Retain originals immutably; compress the payload, never the evidence.

See your workload's compression ratio measured →