The issue of routing individual requests to different models is a developed area; We ourselves at Inanto have been doing this for almost a dozen years. And heck, I'm definitely surprised by OpenRouter's valuation and their sale to Pass. The current work deals with something else - routing individual tokens between different models, for example, according to the "complexity" of the token (entropy). When the entropy is high, we give the generation to a heavier model. When everything is simple, we produce with a small model. The most interesting thing about the job, IMHO, is the huge overhead for all the associated machines if this thing is not optimized. TokenRouter: An Efficient Submission System for LLM Token-Level Routing Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yi Jie, Yu Wang Paper: Code: Model: N/A.
Overview: What they did: The authors presented TokenRouter - an SGLang runtime for step-by-step LLM routing, where the generation switches between small and large models in single tokens. inside -. A three-function API, asynchronous subservers for each model, saving KV cache state between calls, and a lazy batch scheduler on Markov networks. What we got: On 5 AIME2024 reasoning problem routing algorithms and SWE-Smith agent scenarios with 4 parallel requests, the throughput ranged from 76.6 to 290.3 token/s compared to 4.5-63.1 token/s for the flow-based baseline in SGLang-7 for SGLang's official increase to 73. 2.01x–64.15x). On the R2R router, with 8 requests, the speed increased from 134.9 to 372.5 tokens/s (2.76x, of which 1.71x is due to CUDA graphics). With 16 requests, the system outperforms R2R by 3.61x in the same parallel (or 18.58x over R2R with one request, with 1.13x speed per user).