🤖 AI Summary
This work proposes a non-parametric, prompt-level distillation method that balances inference efficiency and interpretability by extracting explicit reasoning paths from large language models and encoding them into structured instructions embedded within the system prompt of a smaller model. Without requiring any fine-tuning, this approach effectively transfers complex reasoning capabilities while introducing negligible latency. Evaluated on StereoSet and Contract-NLI, the method boosts the Macro F1 scores of the Gemma-3 4B model from 57% and 67% to 90.0% and 83%, respectively, achieving state-of-the-art performance. Crucially, the full reasoning logic is preserved in an interpretable form, enabling direct human verification and validation of the model’s decision process.
📝 Abstract
Advanced reasoning typically requires Chain-of-Thought prompting, which is accurate but incurs prohibitive latency and substantial test-time inference costs. The standard alternative, fine-tuning smaller models, often sacrifices interpretability while introducing significant resource and operational overhead. To address these limitations, we introduce Prompt-Level Distillation (PLD). We extract explicit reasoning patterns from a Teacher model and organize them into a structured list of expressive instructions for the Student model's System Prompt. Evaluated on the StereoSet and Contract-NLI datasets using Gemma-3 4B, PLD improved Macro F1 scores from 57\% to 90.0\% and 67\% to 83\% respectively, enabling this compact model to match frontier performance with negligible latency overhead. These expressive instructions render the decision-making process transparent, allowing for full human verification of logic, making this approach ideal for regulated industries such as law, finance, and content moderation, as well as high-volume use cases and edge devices.