Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch

2025-04-17

AI Summary

This work addresses the performance gap between pruning-from-initialization (PaI) and dense-to-sparse training, where PaI suffers from uncontrollable parameter sign initialization. To resolve this, the authors propose Sign-In—a dynamic reparameterization method that enables sign-aware, learnable initialization. Theoretically, Sign-In guarantees gradient-driven sign flipping under optimization; practically, it introduces a sign-determinism-driven gradient update mechanism that preserves sparsity while enabling sign learning. This is the first work to theoretically and empirically demonstrate that sign determinism alone suffices for efficient PaI. Evaluated on ImageNet and other benchmarks, Sign-In significantly improves convergence speed and final accuracy across diverse sparse architectures, closing the gap with dense-to-sparse baselines. The residual performance gap is explicitly attributed to insufficient co-optimization of signs and magnitudes—highlighting a fundamental limitation in current PaI paradigms.

Technology Category

• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Privacy-enhancing technologies

Abstract

The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training presents a major roadblock for efficient deep learning. According to the Lottery Ticket Hypothesis, PaI hinges on finding a problem specific parameter initialization. As we show, to this end, determining correct parameter signs is sufficient. Yet, they remain elusive to PaI. To address this issue, we propose Sign-In, which employs a dynamic reparameterization that provably induces sign flips. Such sign flips are complementary to the ones that dense-to-sparse training can accomplish, rendering Sign-In as an orthogonal method. While our experiments and theory suggest performance improvements of PaI, they also carve out the main open challenge to close the gap between PaI and dense-to-sparse training.

Problem

• Bridging performance gap in sparse neural network training
• Finding correct parameter signs for initialization
• Dynamic reparameterization to enable sign flips

Innovation

• Dynamic reparameterization enables sign flips
• Sign-In complements dense-to-sparse training
• Focuses on parameter initialization signs