LLM routing, KV caches, and misaligned incentives

There's a dominant narrative in AI startup spaces right now that you can seamlessly slash your LLM inference costs by dynamically routing requests to the cheapest model at any given millisecond. The basic pitch usually claims that you should commoditize your LLM stack by sending every prompt to a smart router that evaluates the complexity of the request and dynamically dispatches it to whichever provider is currently offering the lowest spot price.

This is a cocktail-party understanding of LLM economics. It's a viewpoint that completely relies on treating API calls as if they are stateless, 2012-era AWS Lambda functions, completely ignoring the physical reality of how transformer inference actually works at the hardware level.

If we want to understand why treating a stateful transformer as a generic, stateless REST endpoint is a massive ops smell, we have to look at the literal math of how the silicon operates. Based on the documented transformer inference arithmetic proving prefill is compute-bound, token generation is fundamentally memory-bandwidth bound. It's constrained by strict physical limits, like the 3.35 TB/s peak memory bandwidth on an H100 GPU. The prefill phase, however, is massively compute-bound because the engine has to matrix-multiply the entire prompt simultaneously before it can generate a single word.

Because the attention mechanism in standard transformers scales quadratically (O(n^2)) with sequence length, the only physical mechanism preventing unbounded, exponential latency in long-context interactions is the Key-Value (KV) cache. The KV cache stores the massive matrices of intermediate attention states for previously processed tokens in VRAM so they don't have to be recalculated from scratch on every turn. (Alternative architectures that researchers are exploring, like linear attention or Mamba-style state space models, attempt to bypass this entirely, but for our purposes, essentially all production frontier models rely heavily on KV caching and will for the foreseeable future, even though people keep claiming transformers are dying).

When dynamic routing middleware—like the various tokenless routing abstractions that attempt to load-balance inference—decides to jump from a Claude model to a Llama model mid-session to save a fraction of a cent per token, it is effectively destroying that warm KV cache. It forces a cold start on a totally different cluster.

Let's do the arithmetic to see what this actually costs in physical time. If you have a conversation with 10,000 tokens of context, and you route the user's next message to a new provider, you are paying the full time-to-first-token (TTFT) penalty for a cold compute-bound prefill. You are taking [number of context tokens] / [compute limit] and adding it directly to the user's wait time, repeatedly, every single time the router switches providers. And that's just the compute penalty on the GPU. You're also paying the network penalty of opening a completely new connection (DNS resolution, TLS handshake, and basic TCP slow start) to a datacenter that might be in a different region, whereas a sticky session to one provider can often reuse an existing keep-alive connection.

This hardware reality strictly dictates the economic reality, which is why Anthropic's 90% discount on cached input tokens and OpenAI's 50% discount for reusing recently seen tokens exist. The pricing structures deliberately reflect the physical fact that cache hits cost the provider drastically less compute. They are pricing their APIs to aggressively incentivize developers to maintain cache locality and stop trashing memory bandwidth.

By dynamically routing, you are intentionally forfeiting these discounts in exchange for an illusion of frugality.

If you take 10,000 tokens and multiply by a $3.00 per million base rate, the un-cached cost is 10000 * 3.00 / 1000000 = $0.030. When you apply the 90% prompt caching discount, you pay exactly $0.003. If your middleware routes that identical prompt to an allegedly cheaper open-weight model charging $1.00 per million tokens, the lack of a warm KV cache means you pay the full un-cached rate, which is 10000 * 1.00 / 1000000 = $0.010.

You just paid 3.33x more money to make the application objectively slower. That's a ridiculous trade to make, but companies make it constantly.

If you read the marketing materials from routing vendors, the strongest counter-argument you will find is that the latency penalty of a cold prefill is negligible for small contexts, and that the base price gap between frontier models and open-weight models is so large that the math inevitably favors fanning out requests to multiple providers to find the cheapest acceptable response.

That math is entirely backward.

Even if you assume the context window is small enough that the dollar cost is negligible, fanning out requests fundamentally breaks AI hardware state because you are paying the prefill penalty repeatedly across every provider. If you send the same 2,000 tokens to four different providers to evaluate the cheapest output in parallel, you are chewing through 8,000 tokens of compute-bound prefill globally. You are paying the TTFT penalty four times over, discarding the physical mechanism that makes inference computationally viable in the first place. This guarantees that your p99 latency will permanently sit at the floor of a cold start.

Since the math proves that dynamic mid-turn routing is physically slower and economically negative, we have to ask why VC-backed startups keep building this and why engineering managers keep buying it.

The VC pathology is simply funding pattern-matching to old web-app load balancing (which is somewhat ironic given that modern abstractions are designed to hide the concept of a provider entirely, meaning the developer is structurally shielded from knowing when their application is actively self-sabotaging its own cache locality). They assume that if fanning out HTTP requests to stateless microservices worked to save AWS costs in 2012, fanning out LLM prompts must work in 2024. It's a classic case of applying an abstraction to a domain where the underlying physical constraints have completely shifted.

But the demand side is where the systemic failure really crystallizes. This creates an environment defined by severe information asymmetry creating a market for lemons. The buyer—usually an engineering manager or a Director of Platform—is evaluated on metrics that completely diverge from the actual physical experience of the user. The manager is heavily incentivized to optimize blended token costs. If they deploy a dynamic router and the monthly invoice on the finance dashboard drops because 15% of queries were routed to a model with a lower nominal base price, that looks like a massively successful cost-saving OKR on their performance review.

The fact that the router intentionally forfeited the 90% KV cache discount, increased the total global compute expended by fanning out, and degraded the user's response time by two entire seconds is quietly swept under the rug. This happens because TTFT degradation in a multi-provider fan-out setup is extremely difficult to measure precisely on standard APM dashboards. Typical Datadog or New Relic setups aggregate P50 and P99 latencies across a blend of completely different interaction types. The massive penalty of a 10,000-token mid-session context switch gets completely washed out in the statistical noise of thousands of tiny, 50-token zero-shot queries <abbr title="assuming they even bother to instrument latency per interaction type, which they rarely do">that dominate the raw query volume</abbr>.

You end up with a system where everybody locally optimizes for their own dashboard. The routing startup gets to claim they are disrupting OpenAI's margins. The engineering manager gets promoted for lowering the blended average cost per query. Finance is happy because the total invoice looks marginally smaller when you squint at it. The only entity that actually pays the price for this massive pile of burning compute and O(n^2) inefficiency is the end user, who is left staring at a spinning loading wheel, waiting for a remote cluster to needlessly recalculate 10,000 tokens of attention states from scratch because someone decided to pretend they saved a tenth of a cent.

Observers often look at this architectural tire fire and assume the buyers just don't understand the underlying arithmetic, completely missing the fact that these managers are making perfectly rational career decisions given the deeply dysfunctional corporate environment. If you were to ask someone, abstractly, if a company wants to intentionally make their product 3x slower by repeatedly destroying their own cache just to pretend they saved a fraction of a cent per token, I suspect they'd tell you no. But insofar as a company can be said to want anything, it wants what it incentivizes—and right now, it incentivizes managers to optimize for the easily measurable invoice while pushing the unmeasured physical reality onto the user.