By Ailoot
An LLM router can look cheaper in a demo because the test never forces it to recover from a bad model choice. Before sending production traffic through a router, compare the same requests with and without it. A small worksheet is enough to spot expensive fallbacks and tasks that break after the switch.
Build a 24-request replay
Pick six examples from each of four groups that matter to your system. A support summarizer might use short answers, long-context summaries, structured JSON and ambiguous requests. Remove private data. For each example, write one observable pass condition. A JSON answer might need to parse and contain all required fields. A summary might need to preserve a specific decision and its date.
Run each request through your current model and through the candidate router. Keep the prompt, output cap and pass rule fixed. Record the actual model, each fallback attempt, time until the final answer and total cost across attempts. An answer that succeeds on its second try still used two calls.
Measure the cost of an acceptable answer
Make one row per request and one summary per task group. Count accepted answers, add up costs and divide total cost by the number of accepted answers. Keep timeout costs in the numerator. Mark groups with zero accepted answers as failures without calculating a ratio.
Suppose a configuration spends USD 0.12 across six prompts and passes four. It costs USD 0.03 per accepted answer. Another spends USD 0.15 and passes all six, so it costs USD 0.025 per accepted answer. Compare the missed cases before changing the routing rule.
Check warm prompts and fallbacks separately
Run your most common repeated prompts again and label their results as a separate warm-cache run. Record cache hits when your provider reports them. Check whether switching models loses the cache benefit. Review the slowest fallback from each group and decide what latency and cost you can tolerate before a request should fail fast.
Use this replay for a first pass. Set the minimum acceptance rate and maximum fallback delay for each group before a staged rollout, then measure them on a larger traffic sample.
Read the full method in How to Evaluate an LLM Router for Production on Ailoot.