🤖 AI Summary
For linear regression with unknown noise distribution, this paper proposes a data-driven method to construct convex loss functions such that the resulting empirical risk minimizer achieves the minimal asymptotic variance among all convex M-estimators. The key innovation is the first integration of score matching into the semiparametric convex M-estimation framework, unifying log-concave projection of the error density and minimization of Fisher divergence. For non-log-concave errors (e.g., Cauchy), the method automatically yields Huber-type optimal convex losses. We prove that the proposed estimator attains the minimal asymptotic covariance over all convex M-estimators; under Cauchy errors, its relative efficiency versus the oracle MLE exceeds 0.87. The algorithm combines nonparametric estimation of density derivatives, Fisher divergence optimization, and convex programming—ensuring both theoretical optimality and computational efficiency. Numerical experiments confirm its superior finite-sample performance.
📝 Abstract
In the context of linear regression, we construct a data-driven convex loss function with respect to which empirical risk minimisation yields optimal asymptotic variance in the downstream estimation of the regression coefficients. Our semiparametric approach targets the best decreasing approximation of the derivative of the log-density of the noise distribution. At the population level, this fitting process is a nonparametric extension of score matching, corresponding to a log-concave projection of the noise distribution with respect to the Fisher divergence. The procedure is computationally efficient, and we prove that our procedure attains the minimal asymptotic covariance among all convex $M$-estimators. As an example of a non-log-concave setting, for Cauchy errors, the optimal convex loss function is Huber-like, and our procedure yields an asymptotic efficiency greater than 0.87 relative to the oracle maximum likelihood estimator of the regression coefficients that uses knowledge of this error distribution; in this sense, we obtain robustness without sacrificing much efficiency. Numerical experiments confirm the practical merits of our proposal.