Maximum Entropy Least Squares Solutions of Overdetermined Linear Systems

2026-03-15

AI Summary

This work addresses the lack of robustness in ordinary least squares when sparse, large-magnitude outliers are present. The authors propose a maximum-entropy-based weighted least squares framework, wherein weights are interpreted as a discrete probability distribution and determined by maximizing Shannon entropy subject to a mean squared error constraint. This approach automatically downweights outliers while preserving fidelity to inliers. Innovatively treating the mean squared error as a tunable control parameter, the method leverages the principle of maximum entropy to select the least biased weight distribution. The paper establishes the theoretical existence of a locally unique, globally continuable smooth solution branch emanating from the ordinary least squares solution. Further analysis reveals that, in the zero-error limit, the method automatically identifies the largest subset of data consistent with the underlying linear model. Numerical experiments confirm its superior robustness.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

We investigate the theoretical foundations of a recently introduced entropy-based formulation of weighted least squares for the approximation of overdetermined linear systems, motivated by robust data fitting in the presence of sparse gross errors. The weight vector is interpreted as a discrete probability distribution and is determined by maximizing Shannon entropy under normalization and a prescribed mean squared error (MSE) constraint. Unlike classical ordinary least squares, where the error level is an output of the minimization process, here the MSE value plays the role of a control parameter, and entropy selects the least biased weight distribution achieving the prescribed accuracy. The resulting optimization problem is nonconvex due to the nonlinear coupling between the weights and the solution induced by the residual constraint. We analyze the associated optimality system and characterize stationary points through first- and second-order conditions. We prove the existence and local uniqueness of a smooth branch of entropy-maximizing configurations emanating from the ordinary least squares solution and establish its global continuation under suitable nondegeneracy conditions. Furthermore, we investigate the asymptotic regime as the prescribed MSE tends to zero and show that, under appropriate assumptions, the limiting configuration concentrates on a largest subset of data consistent with the linear model, thus suppressing the influence of outliers. Two numerical experiments illustrate the theoretical findings and confirm the robustness properties of the method.

Problem

• overdetermined linear systems
• robust data fitting
• sparse gross errors
• outliers
• maximum entropy

Innovation

• maximum entropy
• weighted least squares
• overdetermined linear systems
• robust estimation
• outlier suppression