New framework optimizes LLM inference costs via adaptive model activation

The study proves that for series-based inference networks, the optimal activation policy follows a threshold structure. For generative tasks, a single confidence threshold determines when to escalate to a costlier model, whereas discriminative tasks require a distinct threshold for each class. The authors provide a structured method to compute these thresholds and offer practical mechanisms for estimating model confidence.

Experiments using open-source LLMs demonstrate that this adaptive routing strategy achieves substantial cost reductions while strictly adhering to specified performance budgets. This work addresses a critical gap in current NLP practices, where multiple expert models are often used in ensemble or series configurations without a principled method for determining the most efficient activation strategy.