Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents

2026-10-06

AI Summary

This study addresses the limitation that small-model agents frequently encounter step failures due to tool-calling errors, while existing query-level routing struggles to accommodate dynamic difficulty shifts during task execution. To this end, this work proposes STEPGATE, a novel uncertainty-aware, step-level dynamic routing framework that precisely escalates high-complexity operations to powerful cloud models, thereby balancing local inference efficiency with cloud performance. Evaluated on the BFCL benchmark, the proposed method achieves a 69.0% success rate on multi-turn tasks while requiring only 30% cloud invocations to substantially narrow the performance gap with stronger models. These results demonstrate that STEPGATE significantly outperforms both random and query-level routing strategies.

Abstract

Small language models (SLMs) are attractive as local agent controllers because they reduce remote inference, latency, and deployment footprint, yet structured tool errors can cause an agent step to fail. Existing routers typically select a model once per query. However, agents expose sequential decision points whose difficulty dynamically changes based on intermediate observations. We propose STEPGATE, an uncertainty-aware handoff framework that scores each local SLM action and selectively escalates challenging steps to a stronger model. On a 52-task held-out single-step BFCL-derived test split, the Qwen2.5-1.5B/7B pair attains 82.7% task success with 30.8% escalation, versus 67.3% local-only and 75.4% random escalation (which uses 33.8% escalation). In a separate multi-turn evaluation, STEPGATE achieves 69.0% trajectory success and 84.0% action success using only 30.0% cloud actions, compared with 48.0%/70.5% local-only, 60.0%/78.2% random escalation, and 57.0%/77.1% query-level routing (strong-only achieves 82.0% trajectory success at 100% cloud actions). These results suggest that step-level escalation recovers a large share of the performance gap to the stronger Qwen2.5-7B backend at a matched cloud-action rate while transmitting fewer tokens remotely. However, our evaluation is limited to one model family, a single stronger backend, and scripted tasks. Furthermore, the test sets are small, multi-turn comparisons rely on paired intervals and statistical tests, and our risk tiers serve as research annotations rather than formal safety guarantees.

Problem

• Small Language Models
• Agent Routing
• Step-level Handoff
• Uncertainty Estimation
• Cloud Escalation

Innovation

• Uncertainty-Aware Handoff
• Step-Level Routing
• Small Language Model Agents
• Selective Escalation
• Cloud-Edge Collaboration