Is Personalized Modality Weighting Actually Personalized? A Controlled Audit of Per-User Weighting Claims in Multimodal Recommenders

2026-08-06

AI Summary

This work investigates whether user-specific modality weighting mechanisms widely adopted in multimodal recommendation systems genuinely capture individual user preferences. To this end, the authors propose an auditing framework comprising two metrics—real-GM and real-shuf—that evaluate personalization efficacy by comparing six personalized weighting methods against global weights and shuffled user-weight assignments, all under a unified collaborative filtering backbone. Experimental results across three short-video and one cross-domain e-commerce dataset reveal that performance gains from most methods stem primarily from increased model capacity rather than authentic user signals, with gating mechanisms often inducing spurious personalization due to their reliance on shared embeddings. Notably, global modality weights already achieve nearly all attainable gains, while personalized weighting shows no consistent improvement; the proposed audit framework effectively identifies architectures that truly encode user-specific patterns.

Abstract

Per-user modality weighting is deployed at billion-user scale in multimodal recommenders, through user modality-strength vectors, attention gates, meta-weight hypernetworks, and low-rank guided weights, each claiming a ranking gain from user-specific modality preference. Yet, to our knowledge, prior evaluations do not isolate a genuinely user-specific signal from a global modality weight plus model capacity. We audit this family with a two-contrast audit principle, reducing six implementations onto one shared collaborative backbone and measuring a utility gap (real-GM) against a single global modality weight and an identifiability gap (real-shuf) against an eval-time permutation of the user-weight binding. Across three independent short-video corpora, a single global weight already delivers nearly all of the content gain (+1.9/+3.6/+3.5pp over a no-modality baseline, p < .001). Making the weight per-user adds no consistent utility: no implementation wins on all corpora and metrics, and the few positive gaps are small (<=0.9pp) and flip. The shuffle control is necessary but not sufficient, since real-shuf reaches +128% of the content gain for heads that simultaneously lose to the global weight. We trace this dissociation to gates reading the shared collaborative embedding: decoupling the gate input collapses the inflated real-shuf to near zero while the utility conclusion stands. A monotone signal-implant dose-response (capture AUROC rising from 0.57 to 0.89 and from 0.64 to 1.00) verifies the harness would detect user-specific structure if present, and every finding replicates on a fourth, cross-domain e-commerce corpus. We propose reporting real-GM alongside real-shuf as a minimum evidentiary standard for personalization claims.

Problem

• personalized modality weighting
• multimodal recommenders
• user-specific preference
• global modality weight
• personalization audit

Innovation

• personalized modality weighting
• controlled audit
• multimodal recommenders
• user-specific signal
• two-contrast audit