Coordinate Descent Algorithm for Least Absolute Deviations Regression

2026-03-19

AI Summary

This work proposes an efficient coordinate descent algorithm for least absolute deviation (LAD) regression in high-dimensional settings, where conventional solvers suffer from high computational cost and limited scalability. By reformulating each coordinate update as a univariate (weighted) median problem that admits a closed-form solution, the method avoids matrix inversion and is naturally suited to scenarios with $p \geq n$. It represents the first effective application of coordinate descent to LAD regression, leveraging the convexity of the objective function and the analytical tractability of the median to handle non-differentiability while ensuring stable convergence—without reliance on specialized optimization libraries. Experiments on both synthetic and real-world datasets demonstrate that the proposed approach achieves accuracy comparable to linear programming solvers while offering substantially improved computational efficiency and numerical stability in high-dimensional regimes.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Least Absolute Deviations (LAD) regression provides a robust alternative to ordinary least squares by minimizing the sum of absolute residuals. However, its widespread use has been limited by the computational cost of existing solvers, particularly simplex-based methods in high-dimensional settings. We propose a coordinate descent algorithm for LAD regression that avoids matrix inversion, naturally accommodates the non-differentiability of the objective function, and remains well-defined even when the number of predictors exceeds the number of observations. The key observation is that each coordinate update reduces to a one-dimensional minimization admitting a closed-form solution given by a median or weighted median. The resulting algorithm has per-iteration complexity $O(p\,n \log n)$ and is provably convergent due to the convexity of the LAD objective and the exactness of each coordinate update. Experiments on synthetic and real datasets show that the method matches the accuracy of linear-programming-based LAD solvers while offering improved scalability and stability in high-dimensional regimes, including cases where $p \ge n$. The method is easy to implement, requires no specialized optimization software, and provides a practical tool for robust linear models.

Problem

• Least Absolute Deviations
• coordinate descent
• high-dimensional regression
• computational cost
• robust regression

Innovation

• Coordinate Descent
• Least Absolute Deviations
• Weighted Median
• High-dimensional Regression
• Robust Regression