The rental car paradox of LLM routing
Oracle routing wastes compute by assuming impossible foreknowledge, so the industry needs prompt-time actuaries that score risk before a single token fires.
By Theo Marsh
Sparked by Kimi K3 Is Competitive with Fable; Kimi K3 and Fable Is SoTA · discussion
Fireworks AI released a benchmark yesterday comparing Kimi K3 to Fable, and a subsequent debate over on Hacker News highlighted exactly how warped our mental model of LLM routing has become. Everyone in these discussions eventually points to "oracle routing" as the gold-standard baseline for cost savings. The math always looks undeniably spectacular on paper.. right up until it meets a production pipeline.
But that math relies on a fundamental paradox. It becomes obvious the moment you step away from a static evaluation dataset and map the problem to a physical rental car counter.
Imagine you're standing at the airport counter, exhausted after a flight. The agent asks if you want the collision damage waiver. According to the fine print, you can pay $15 to $30 a day for comprehensive coverage. Calling a massive, highly capable frontier model like GPT-4 or Claude 3.5 Sonnet is buying that waiver. You pay a premium upfront for a near-guarantee that your vehicle returns intact, regardless of what the road throws at you.
Or you decline coverage. You save the daily fee entirely, but you assume the risk of a $5,000 out-of-pocket charge if you scrape a concrete pillar in the parking garage. Routing a complex task to a hyper-cheap, smaller open-source model is declining the coverage. You save a mountain of compute, right up until the model hallucinates a parameter and catastrophically breaks your automated downstream workflow.
So here is where the "oracle" absurdity creeps in.
In standard industry implementations -- like the methodology detailed in the RouteLLM release -- an oracle router relies on ground-truth labels. It evaluates historical performance data to definitively pick the cheaper model whenever that model is mathematically capable of answering correctly.
Translated back to the rental counter, the oracle baseline assumes you can confidently decline the insurance waiver on Tuesday because you possess absolute foreknowledge that you will drive safely, and then purchase it on Wednesday because you know you will rear-end a Honda Accord at exactly 3:15 PM. It requires absolute foreknowledge of the crash.
Let's do the literal arithmetic to prove this paradox in production.
Suppose you receive an inbound prompt and want to route it optimally to save compute. To know if the cheap model generates the correct response, you actually have to run it. Let's say you route 1,000,000 tokens through this evaluation system. The cheap model boasts a massive 50x cost reduction compared to the frontier model. You generate the output. Now you have to verify it against a rubric.
If the cheap model fails, you must discard that generated text entirely, spin up the expensive frontier model, and run the identical prompt a second time.
You are literally paying to run both models in a convoluted attempt to save money on one.
And worse, as Edward Benson pointed out regarding agent performance, the mathematical reality of multiplying probabilities means that compounding errors from cheaper models become impossible to ignore over long horizons. A smaller model crashing on step four of a multi-step chain doesn't just cost you the redo of step four. It poisons the entire context window, dragging the probability of a successful final outcome toward zero.
This mental model of benchmarking against an impossible future-state evaluation is a dead end. We should abandon the assumption that knowing the answer ahead of time is a valid baseline for routing efficiency.
What the industry actually needs to build is a Prompt-Time Actuary.
Actuaries calculate risk exclusively off historical tables, long before you hand them the keys. In a routing context, this component scores the incoming request purely on syntax, structural complexity, or a tiny heuristic classifier. It assigns a risk penalty before a single generation token fires.
The 'Oracle' Paradox:
Prompt -> Run Cheap Model -> Evaluate -> Run Expensive Model
The Prompt-Time Actuary:
Prompt -> Actuary Risk Score -> Pick One Model
We can ground this in a highly specific workflow.
A user submits a prompt asking a system to extract names from a provided text document and format them as a 5-item JSON array. The actuary parses the request, flags the presence of the word "JSON" alongside a simple extraction verb, and assigns a low risk score. It declines the coverage. The system routes the prompt directly to the cheap model.
Then the next user submits a prompt asking the system to cross-reference three conflicting legal statutes and summarize the prevailing liability limit. The actuary identifies deep semantic complexity and conflicting logical constraints. High risk. It buys the waiver. The system routes the prompt directly to the frontier model.
The actuary prices the historical probability of failure and accepts the upfront premium, completely bypassing the compute cost of an experimental run.
My guess? In 5 years, the models themselves will be completely commoditized. The only real moat left post-AGI will be the routing layer that sits just above them.