Sparse Coding Representation of 2-way Data

2025-09-12

AI Summary

To address poor dictionary generalizability, high sample complexity, and insufficient structural modeling in 2D signal sparse coding, this paper proposes a low-rank sparse coding framework for joint spatiotemporal modeling. Methodologically, it introduces a learnable low-rank encoding constraint, constructs an analysis-synthesis hybrid dictionary model, and develops the AODL convex relaxation algorithm—proven theoretically to yield solutions equivalent to the original nonconvex problem, efficiently solved via convergent alternating optimization. Key contributions include: (i) substantially reduced sample complexity; (ii) enhanced dictionary interpretability and cross-sample generalization; and (iii) up to 90% sparsity gain on both synthetic and real-world data, outperforming fixed-dictionary and non-low-rank baselines in reconstruction accuracy while maintaining robust missing-value imputation performance.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Sparse dictionary coding represents signals as linear combinations of a few dictionary atoms. It has been applied to images, time series, graph signals and multi-way spatio-temporal data by jointly employing temporal and spatial dictionaries. Data-agnostic analytical dictionaries, such as the discrete Fourier transform, wavelets and graph Fourier, have seen wide adoption due to efficient implementations and good practical performance. On the other hand, dictionaries learned from data offer sparser and more accurate solutions but require learning of both the dictionaries and the coding coefficients. This becomes especially challenging for multi-dictionary scenarios since encoding coefficients correspond to all atom combinations from the dictionaries. To address this challenge, we propose a low-rank coding model for 2-dictionary scenarios and study its data complexity. Namely, we establish a bound on the number of samples needed to learn dictionaries that generalize to unseen samples from the same distribution. We propose a convex relaxation solution, called AODL, whose exact solution we show also solves the original problem. We then solve this relaxation via alternating optimization between the sparse coding matrices and the learned dictionaries, which we prove to be convergent. We demonstrate its quality for data reconstruction and missing value imputation in both synthetic and real-world datasets. For a fixed reconstruction quality, AODL learns up to 90% sparser solutions compared to non-low-rank and analytical (fixed) dictionary baselines. In addition, the learned dictionaries reveal interpretable insights into patterns present within the samples used for training.

Problem

• Developing low-rank coding model for 2-dictionary sparse representation
• Establishing sample complexity bounds for dictionary generalization
• Solving convex relaxation via convergent alternating optimization

Innovation

• Low-rank coding model for two-dictionary learning
• Convex relaxation solution via alternating optimization
• Sparse representation with interpretable pattern insights