FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning

2026-01-26

AI Summary

This work addresses the high computational cost and limited performance of large language model (LLM) reasoning, often caused by redundant and unreliable reasoning paths. The study introduces the novel concept of “reasoning anomalies” and proposes an attention-driven, sentence-level anomaly detection and pruning mechanism that significantly enhances both efficiency and reliability while preserving reasoning capability. By analyzing attention weights to identify and eliminate anomalous paths, the method produces shorter and more trustworthy reasoning traces. Experimental results across four benchmarks demonstrate an average reduction of 69.68% in token usage, a 26.70% improvement in accuracy, and a substantial decrease in attention anomalies—evidenced by a 15.97% drop in maximum infinity norm and a 91.09% reduction in average kurtosis.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We propose FROST, an attention-aware method for efficient reasoning. Unlike traditional approaches, FROST leverages attention weights to prune uncritical reasoning paths, yielding shorter and more reliable reasoning trajectories. Methodologically, we introduce the concept of reasoning outliers and design an attention-based mechanism to remove them. Theoretically, FROST preserves and enhances the model's reasoning capacity while eliminating outliers at the sentence level. Empirically, we validate FROST on four benchmarks using two strong reasoning models (Phi-4-Reasoning and GPT-OSS-20B), outperforming state-of-the-art methods such as TALE and ThinkLess. Notably, FROST achieves an average 69.68% reduction in token usage and a 26.70% improvement in accuracy over the base model. Furthermore, in evaluations of attention outlier metrics, FROST reduces the maximum infinity norm by 15.97% and the average kurtosis by 91.09% compared to the base model. Code is available at https://github.com/robinzixuan/FROST

Problem

• reasoning outliers
• efficient reasoning
• attention mechanism
• token reduction
• reasoning reliability

Innovation

• reasoning outliers
• attention-based pruning
• efficient reasoning
• token reduction
• reliable reasoning trajectories