Learning from the Future: Privileged Self-Distillation for Sequential Recommendation

2026-07-29

AI Summary

This work addresses the limitation of conventional sequential recommendation models, which rely solely on one-hot labels of the next item, resulting in sparse supervision and an inability to capture users’ relative preferences toward non-target items. To overcome this, the authors propose a Privileged Self-Distillation (PSD) framework that leverages future user interactions as privileged information during training. PSD employs a dual-attention masking mechanism with a shared backbone to disentangle historical and future contexts, and integrates an advantage-reachability gating module with a momentum-based teacher model to generate soft labels for self-distillation. The framework enables end-to-end, single-stage training without requiring a pre-trained teacher and incurs no additional inference overhead. Extensive experiments demonstrate that PSD consistently outperforms state-of-the-art methods across multiple benchmark datasets and various backbone architectures, including Transformers.

Abstract

Sequential recommenders are commonly trained with one-hot next-item labels under a causal (prefix-only) objective aligned with inference. While deployment-compatible, this supervision offers little insight into relative preferences among non-target items. Yet logged interaction sequences contain an additional supervisory source: interactions following the target often reveal how user intent evolves, making the target easier to interpret. We treat these future interactions as training-only privileged information, available during learning but not at inference. This raises a natural question: can future interactions provide richer supervision while keeping training aligned with inference-time prediction? We propose Privileged Self-Distillation (PSD), a framework that separates learning-time information from inference-time information. PSD applies two attention masks to the same backbone: a future-aware view yields a privileged teacher distribution conditioned on past and future interactions, while a prefix-only view yields the student distribution used for deployment. Distilling the privileged distribution converts future interactions into training-only supervision rather than inference-time inputs. Since both views share a backbone, the teacher's advantage is purely informational, not architectural, removing the need for a separately pretrained teacher and letting its supervision adapt as the student evolves. PSD further uses an advantage-reachability gate to focus distillation on teacher signals likely supported by the observed prefix, along with a momentum-averaged teacher for stable targets. The framework is optimized end-to-end in a single stage, leaving the deployed model and inference cost unchanged. Experiments across public benchmarks and diverse backbones show consistent improvements.

Problem

• Sequential Recommendation
• Privileged Information
• Future Interactions
• Supervision Signal
• Inference Alignment

Innovation

• Privileged Self-Distillation
• Sequential Recommendation
• Future-aware Supervision
• Knowledge Distillation
• Causal Inference Alignment