Compressed Newton-direction-based Thresholding Methods for Sparse Optimization Problems

2025-10-05

AI Summary

To address the high computational cost and difficulty in balancing convergence efficiency of thresholding-based Newton-type methods for sparse optimization, this paper proposes a Compressed Newton Thresholding Optimization framework. It compresses the Newton direction onto a low-dimensional subspace and incorporates diagonal regularization, yielding two novel algorithms: Compressed Newton Hard Thresholding Pursuit (CNHTP) and Compressed Newton Optimal Thresholding Pursuit (CNOTP). Under the Restricted Isometry Property (RIP), we establish rigorous theoretical guarantees of global convergence. The algorithms integrate compressed direction computation, adaptive thresholding selection, and regularization, preserving Newton-type convergence rates while significantly reducing per-iteration complexity. Experiments demonstrate that the proposed methods match state-of-the-art algorithms in recovery success rate and solution accuracy, while achieving superior computational efficiency and robustness.

Technology Category

• Search and Optimization: Non-convex Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Thresholding algorithms for sparse optimization problems involve two key components: search directions and thresholding strategies. In this paper, we use the compressed Newton direction as a search direction, derived by confining the classical Newton step to a low-dimensional subspace and embedding it back into the full space with diagonal regularization. This approach significantly reduces the computational cost for finding the search direction while maintaining the efficiency of Newton-like methods. Based on this new search direction, we propose two major classes of algorithms by adopting hard or optimal thresholding: the compressed Newton-direction-based thresholding pursuit (CNHTP) and compressed Newton-direction-based optimal thresholding pursuit (CNOTP). We establish the global convergence of the proposed algorithms under the restricted isometry property. Experimental results demonstrate that the proposed algorithms perform comparably to several state-of-the-art methods in terms of success frequency and solution accuracy for solving the sparse optimization problem.

Problem

• Developing efficient thresholding algorithms for sparse optimization problems
• Reducing computational cost while maintaining Newton-method efficiency
• Establishing global convergence under restricted isometry property conditions

Innovation

• Compressed Newton direction reduces computational cost
• Hard and optimal thresholding strategies for sparsity
• Global convergence under restricted isometry property