Structure-Preserving Margin Distribution Learning for High-Order Tensor Data with Low-Rank Decomposition

2025-09-17

AI Summary

To address the structural distortion and high computational cost caused by vectorization in high-dimensional tensor classification, this paper proposes the first structure-preserving margin distribution learning framework specifically designed for tensor spaces. Methodologically, we extend the Large Margin Distribution Machine (LMDM) to the tensor domain by parameterizing the weight tensor via low-rank CP and Tucker decompositions, thereby preserving the intrinsic multilinear structure. We jointly optimize the margin distribution using first- and second-order gradient statistics and develop a dual-gradient alternating descent algorithm to update factor matrices and the core tensor. Experiments on diverse tensor datasets—including MNIST, natural images, and fMRI—demonstrate that our approach significantly outperforms baseline methods such as SVM, vectorized LMDM, and Support Tensor Machines. Notably, the Tucker-based variant achieves the highest classification accuracy, while exhibiting superior robustness and generalization capability.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The Large Margin Distribution Machine (LMDM) is a recent advancement in classifier design that optimizes not just the minimum margin (as in SVM) but the entire margin distribution, thereby improving generalization. However, existing LMDM formulations are limited to vectorized inputs and struggle with high-dimensional tensor data due to the need for flattening, which destroys the data's inherent multi-mode structure and increases computational burden. In this paper, we propose a Structure-Preserving Margin Distribution Learning for High-Order Tensor Data with Low-Rank Decomposition (SPMD-LRT) that operates directly on tensor representations without vectorization. The SPMD-LRT preserves multi-dimensional spatial structure by incorporating first-order and second-order tensor statistics (margin mean and variance) into the objective, and it leverages low-rank tensor decomposition techniques including rank-1(CP), higher-rank CP, and Tucker decomposition to parameterize the weight tensor. An alternating optimization (double-gradient descent) algorithm is developed to efficiently solve the SPMD-LRT, iteratively updating factor matrices and core tensor. This approach enables SPMD-LRT to maintain the structural information of high-order data while optimizing margin distribution for improved classification. Extensive experiments on diverse datasets (including MNIST, images and fMRI neuroimaging) demonstrate that SPMD-LRT achieves superior classification accuracy compared to conventional SVM, vector-based LMDM, and prior tensor-based SVM extensions (Support Tensor Machines and Support Tucker Machines). Notably, SPMD-LRT with Tucker decomposition attains the highest accuracy, highlighting the benefit of structure preservation. These results confirm the effectiveness and robustness of SPMD-LRT in handling high-dimensional tensor data for classification.

Problem

• Extends margin distribution learning to high-order tensor data
• Preserves multi-dimensional structure without vectorization
• Uses low-rank decomposition for efficient tensor classification

Innovation

• Direct tensor processing without vectorization
• Low-rank decomposition for weight parameterization
• Alternating optimization with double-gradient descent