Towards The Implicit Bias on Multiclass Separable Data Under Norm Constraints

2026-03-24

AI Summary

This work investigates the implicit bias induced by gradient-based optimization algorithms when training over-parameterized models on multi-class separable data, and its impact on generalization. Building upon the normalized steepest descent (NSD) framework, the authors propose NucGD, a geometry-aware optimizer that explicitly promotes low-rank structure through nuclear norm constraints, and establish its theoretical connection to low-rank projection methods. They further derive an efficient update rule that avoids explicit singular value decomposition (SVD), enabling scalable training via asynchronous power iteration. Both theoretical analysis and empirical results demonstrate that the optimization geometry steers solutions toward maximum-margin directions, and that gradient noise in stochastic optimization plays a critical role in shaping convergence trajectories and implicit bias.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Implicit bias induced by gradient-based algorithms is essential to the generalization of overparameterized models, yet its mechanisms can be subtle. This work leverages the Normalized Steepest Descent} (NSD) framework to investigate how optimization geometry shapes solutions on multiclass separable data. We introduce NucGD, a geometry-aware optimizer designed to enforce low rank structures through nuclear norm constraints. Beyond the algorithm itself, we connect NucGD with emerging low-rank projection methods, providing a unified perspective. To enable scalable training, we derive an efficient SVD-free update rule via asynchronous power iteration. Furthermore, we empirically dissect the impact of stochastic optimization dynamics, characterizing how varying levels of gradient noise induced by mini-batch sampling and momentum modulate the convergence toward the expected maximum margin solutions.Our code is accessible at: https://github.com/Tsokarsic/observing-the-implicit-bias-on-multiclass-seperable-data.

Problem

• implicit bias
• multiclass separable data
• norm constraints
• optimization geometry
• overparameterized models

Innovation

• NucGD
• nuclear norm constraint
• low-rank structure
• SVD-free update
• implicit bias