Multi-Attribute Multi-Grained Adaptation of Pre-Trained Language Models for Text Understanding from Bayesian Perspective

2025-03-08

AI Summary

The impact mechanism of non-IID text data on pre-trained language models (PLMs) remains poorly understood. Method: This paper introduces, for the first time, a Bayesian inference framework for modeling PLM adaptability under non-IID conditions, proposing the Multi-Attribute Multi-Granularity Adaptive (M2A) framework. M2A jointly models semantic, stylistic, and domain-specific attributes alongside lexical, sentential, and document-level granularities, leveraging hierarchical attention, gradient-isolated fine-tuning, and uncertainty calibration to achieve lightweight, uncertainty-aware adaptation. Contribution/Results: Extensive experiments demonstrate that M2A significantly outperforms strong baselines across mainstream text understanding tasks. Under implicit non-IID settings and with large-parameter PLMs, it yields consistent gains of 2.1–4.7 percentage points while increasing inference overhead by less than 3%.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Current neural networks often employ multi-domain-learning or attribute-injecting mechanisms to incorporate non-independent and identically distributed (non-IID) information for text understanding tasks by capturing individual characteristics and the relationships among samples. However, the extent of the impact of non-IID information and how these methods affect pre-trained language models (PLMs) remains unclear. This study revisits the assumption that non-IID information enhances PLMs to achieve performance improvements from a Bayesian perspective, which unearths and integrates non-IID and IID features. Furthermore, we proposed a multi-attribute multi-grained framework for PLM adaptations (M2A), which combines multi-attribute and multi-grained views to mitigate uncertainty in a lightweight manner. We evaluate M2A through prevalent text-understanding datasets and demonstrate its superior performance, mainly when data are implicitly non-IID, and PLMs scale larger.

Problem

• Impact of non-IID information on pre-trained language models.
• Bayesian perspective for integrating non-IID and IID features.
• Multi-attribute multi-grained framework for PLM adaptation.

Innovation

• Bayesian perspective integrates non-IID and IID features
• Multi-attribute multi-grained framework (M2A) for PLM adaptations
• Lightweight method mitigates uncertainty in text understanding