Probabilistic and nonlinear compressive sensing

2025-09-18

AI Summary

This work addresses optimal subset selection and teacher network parameter recovery in probabilistic and nonlinear compressive sensing. We propose a smoothed probabilistic ℓ₀-regularized regression framework. Our method enables exact gradient computation without Monte Carlo sampling—achieving significantly accelerated convergence—and integrates normal-form algorithms to handle parameter symmetries. We further identify and analyze the “rebound effect” and symmetry-induced limitations unique to nonlinear settings, clarifying their fundamental distinction from linear counterparts. Experiments demonstrate that our approach consistently outperforms iterative hard thresholding (IHT) and Lasso-based methods across diverse signal-to-noise ratios. While compression reduces test error, it fails to achieve exact parameter recovery; notably, training loss reduction may coincide with unexpected divergence during optimization.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We present a smooth probabilistic reformulation of $ell_0$ regularized regression that does not require Monte Carlo sampling and allows for the computation of exact gradients, facilitating rapid convergence to local optima of the best subset selection problem. The method drastically improves convergence speed compared to similar Monte Carlo based approaches. Furthermore, we empirically demonstrate that it outperforms compressive sensing algorithms such as IHT and (Relaxed-) Lasso across a wide range of settings and signal-to-noise ratios. The implementation runs efficiently on both CPUs and GPUs and is freely available at https://github.com/L0-and-behold/probabilistic-nonlinear-cs. We also contribute to research on nonlinear generalizations of compressive sensing by investigating when parameter recovery of a nonlinear teacher network is possible through compression of a student network. Building upon theorems of Fefferman and Markel, we show theoretically that the global optimum in the infinite-data limit enforces recovery up to certain symmetries. For empirical validation, we implement a normal-form algorithm that selects a canonical representative within each symmetry class. However, while compression can help to improve test loss, we find that exact parameter recovery is not even possible up to symmetries. In particular, we observe a surprising rebound effect where teacher and student configurations initially converge but subsequently diverge despite continuous decrease in test loss. These findings indicate fundamental differences between linear and nonlinear compressive sensing.

Problem

• Improving convergence speed for probabilistic compressive sensing
• Comparing performance against existing compressive sensing algorithms
• Investigating parameter recovery limits in nonlinear compressive sensing

Innovation

• Smooth probabilistic reformulation of l0 regression
• Efficient CPU and GPU implementation for rapid convergence
• Investigates nonlinear compressive sensing with symmetry analysis