Distil-xLSTM: Learning Attention Mechanisms through Recurrent Structures

2025-03-24

AI Summary

This work addresses the challenge of knowledge distillation from Transformer-based large language models (LLMs) to recurrent architectures (e.g., xLSTM, Mamba), stemming from their structural heterogeneity with self-attention mechanisms. We propose Distil-xLSTM—the first compact language model built entirely on a recurrent architecture that explicitly models and distills the parameterization of Transformer attention. Our method leverages xLSTM’s sequential mixing capability to learn a compact representation of attention behavior from an LLM via knowledge distillation, eliminating the conventional requirement for architectural homogeneity. Key contributions include the first interpretable, parameterized approximation of attention mechanisms by a purely recurrent model. Experiments demonstrate that Distil-xLSTM achieves performance competitive with larger models under minimal training overhead, while exhibiting superior computational efficiency and scalability compared to attention-based counterparts—validating the feasibility and effectiveness of recurrent models in learning attention representations.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The current era of Natural Language Processing (NLP) is dominated by Transformer models. However, novel architectures relying on recurrent mechanisms, such as xLSTM and Mamba, have been proposed as alternatives to attention-based models. Although computation is done differently than with the attention mechanism mechanism, these recurrent models yield good results and sometimes even outperform state-of-the-art attention-based models. In this work, we propose Distil-xLSTM, an xLSTM-based Small Language Model (SLM) trained by distilling knowledge from a Large Language Model (LLM) that shows promising results while being compute and scale efficient. Our Distil-xLSTM focuses on approximating a transformer-based model attention parametrization using its recurrent sequence mixing components and shows good results with minimal training.

Problem

• Proposes Distil-xLSTM as efficient alternative to Transformers
• Explores recurrent mechanisms to approximate attention parametrization
• Demonstrates compute-efficient SLM via LLM knowledge distillation

Innovation

• Distil-xLSTM uses recurrent sequence mixing components
• Knowledge distillation from LLM to SLM
• Approximates transformer attention with xLSTM