š¤ AI Summary
This study addresses the scalability limitations of Transformers in long-context modeling, where their O(NĀ²) computational complexity becomes prohibitive. For the first time, it systematically evaluates the performance of the Mamba state space model against the LLaMA Transformer on real-world psychotherapy dialogue data across multi-scale context lengths (512ā8192 tokens). The comparison quantifies the advantages of state space models along two dimensions: computational efficiency (memory footprint and inference latency) and representational efficiency (hidden state dynamics and attention patterns). Results demonstrate that Mamba substantially reduces computational overhead while preserving effective semantic representation under specific conditions, offering empirical evidence and practical guidance for model selection and deployment in long-context applications.
š Abstract
State Space Models (SSMs) have emerged as a promising alternative to Transformers for long-context sequence modeling, offering linear $O(N)$ computational complexity compared to the Transformer's quadratic $O(N^2)$ scaling. This paper presents a comprehensive benchmarking study comparing the Mamba SSM against the LLaMA Transformer on long-context sequences, using dyadic therapy sessions as a representative test case. We evaluate both architectures across two dimensions: (1) computational efficiency, where we measure memory usage and inference speed from 512 to 8,192 tokens, and (2) representational efficiency, where we analyze hidden state dynamics and attention patterns. Our findings provide actionable insights for practitioners working with long-context applications, establishing precise conditions under which SSMs offer advantages over Transformers.