🤖 AI Summary
This work addresses the problem of excessively large neural tangent kernel (NTK) variance in implicit neural representations (INRs), which leads to slow convergence, severe spectral bias, and poor recovery of high-frequency details. We first establish a closed-form decomposition framework for NTK variance, revealing its dependence on pairwise input similarity and scaling terms. Based on this analysis, we provide a unified theoretical explanation for how architectural components—including positional encoding, spherical normalization, and Hadamard modulation—jointly suppress NTK variance and improve condition number. Through rigorous NTK-theoretic analysis, spectral bias diagnostics, and multi-task reconstruction experiments (e.g., image and signed distance function reconstruction), we demonstrate that our variance-regulation mechanism significantly enhances training stability and convergence speed, while achieving superior reconstruction fidelity—particularly in capturing high-frequency structural details.
📝 Abstract
Implicit Neural Representations (INRs) often converge slowly and struggle to recover high-frequency details due to spectral bias. While prior work links this behavior to the Neural Tangent Kernel (NTK), how specific architectural choices affect NTK conditioning remains unclear. We show that many INR mechanisms can be understood through their impact on a small set of pairwise similarity factors and scaling terms that jointly determine NTK eigenvalue variance. For standard coordinate MLPs, limited input-feature interactions induce large eigenvalue dispersion and poor conditioning. We derive closed-form variance decompositions for common INR components and show that positional encoding reshapes input similarity, spherical normalization reduces variance via layerwise scaling, and Hadamard modulation introduces additional similarity factors strictly below one, yielding multiplicative variance reduction. This unified view explains how diverse INR architectures mitigate spectral bias by improving NTK conditioning. Experiments across multiple tasks confirm the predicted variance reductions and demonstrate faster, more stable convergence with improved reconstruction quality.