Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

AI 中文解读

1. 核心亮点：AI路由终于开始“算时间”了——在保证回答质量和省钱的同时，还能让用户少等几秒，效率提升高达40%。 2. 通俗解读：现在企业用大模型时，经常要把不同问题分配给不同的AI模型来节省成本。但以前的路由系统只关心“哪个模型答得好”和“哪个模型更便宜”，完全不考虑用户等待时间。比如你问一个问题，可能被分到一个很慢的模型上，结果等半天才出结果。这篇研究给路由系统装了个“时间预估器”，能提前算好每个模型现在有多忙、你的问题有多长、大概要等多久才能拿到第一个字。然后系统会综合等待时间、回答质量和费用，帮你选最合适的模型，就像打车软件同时考虑价格、距离和等待时间一样。 3. 实际影响：以后你用AI助手时，不会再遇到“明明选了高级模型却卡半天”的尴尬。不管是写论文、查资料还是处理工作，系统都能在你几乎无感的时间内给出最划算的答案。企业也能在降低成本的同时，让客户体验更流畅，比如客服机器人回复更快、智能写作工具出稿更及时。简单说，就是AI用起来更“懂事”了，又快又好还不贵。

arXiv:2607.18253v1 Announce Type: new Abstract: Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost. However, current query routers are largely latency-agnostic and do not consider the generation latency experienced by queries at model instances. In practice, latency is often controlled by load-balancing policies such as round-robin or join-the-shortest-queue, which do not account for model accuracy or inference cost. Incorporating query latency into routing is challenging as it depends not only on the query's prompt length, but also on the current prefill and decode workload at the model instance and the scheduling and batching policy of the serving framework. We design a lightweight latency estimator that simulates autoregressive token batch processing in the serving framework and estimates the time-to-first-token (TTFT) of queries. We incorporate this latency estimator into a latency-aware router that jointly optimizes latency, accuracy, and cost when assigning queries to model instances. Our experimental results indicate that this joint optimization yields up to 40% improvement in accuracy--cost utility while maintaining the same latencies as standard load-balancing approaches.