Equivariant Music Transformer

2026-08-04

AI Summary

Standard Music Transformers often over-rely on absolute positional and pitch representations due to a lack of equivariance under temporal shifts and pitch transpositions, thereby overlooking shared structural patterns. This work introduces equivariance as an explicit inductive bias into music generation for the first time, proposing a Transformer-based self-distillation framework that jointly optimizes next-token prediction with a novel equivariance-preserving regularization loss—without requiring data augmentation or handcrafted features. The proposed approach substantially enhances model equivariance, generalization, and generation quality, consistently outperforming current state-of-the-art methods across both objective metrics and subjective listening evaluations.

Abstract

Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music's translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.

Problem

• equivariance
• music transformer
• representation learning
• pitch transposition
• time shift

Innovation

• equivariance
• music transformer
• self-distillation
• inductive bias
• representation learning