TriAlignGR: Triangular Multitask Alignment with Multimodal Deep Interest Mining for Generative Recommendation

2026-05-05

AI Summary

This work addresses the challenges of semantic content degradation (SCD) and semantic opacity (SSO) in generative recommendation systems that rely on semantic IDs (SIDs). To overcome these limitations, the authors propose TriAlignGR, a unified multitask multimodal framework that, for the first time, integrates visual semantics and deep user intent into SID construction. Through a two-stage multimodal semantic propagation process, TriAlignGR achieves triangular alignment among text, images, and SIDs. Its core innovations include cross-modal semantic alignment, multimodal deep interest mining, and a triangular multitask training strategy, leveraging vision-language models, chain-of-thought reasoning with large language models, and an autoregressive generation architecture. The framework supports eight joint generation tasks—including two novel visual-semantic tasks—without requiring task-specific modules or complex loss weighting. Experiments demonstrate significant improvements in semantic consistency, interpretability, and generalization performance for generative recommendations.

Abstract

We introduce TriAlignGR, a unified multitask-multimodal framework for generative recommendation that establishes two-stage multimodal semantic propagation: (i) encoding visual semantics directly into SIDs via multimodal embeddings, and (ii) enabling the model to decode these semantics through visual description tasks. Existing Semantic ID (SID) pipelines suffer from two fundamental but underexplored problems: \textbf{SID Content Degradation (SCD)}, where cascaded encoding and residual quantization discard critical multimodal and interest-level semantics; and \textbf{SID Semantic Opacity (SSO)}, where models autoregressively generate SID sequences without truly comprehending their underlying meaning, leading to hallucination and poor generalization. Prior work addresses at most text-SID alignment, leaving visual semantics and latent user interests entirely unexploited. TriAlignGR resolves both problems through three tightly integrated components: (1)~\textbf{Cross-Modal Semantic Alignment (CMSA)} integrates visual content into SID construction through both VLM-generated textual descriptions and a multimodal embedding model that directly encodes image features alongside text, ensuring that SIDs inherently carry multimodal semantics; (2)~\textbf{Multimodal Deep Interest Mining (MDIM)} leverages LLM Chain-of-Thought reasoning to extract latent user intents (\eg ``productivity-focused lifestyle'' from noise-canceling headphones) beyond surface attributes, enriching SID semantics before discretization; and (3)~\textbf{Triangular Multitask (TMT)} jointly trains on eight complementary generation tasks under a single autoregressive loss -- including two novel visual-semantic tasks (VisDesc$\to$SID, VisDesc$\to$Title) that map VLM-generated image descriptions to SIDs and titles, completing the SID-Text-Image triangle -- without requiring task-specific towers or complex loss weighting.

Problem

• Semantic ID
• Multimodal Recommendation
• Content Degradation
• Semantic Opacity
• Generative Recommendation

Innovation

• Triangular Multitask Learning
• Multimodal Deep Interest Mining
• Semantic ID
• Cross-Modal Alignment
• Generative Recommendation