STAF: Sinusoidal Trainable Activation Functions for Implicit Neural Representation

2025-02-02

AI Summary

Implicit neural representations suffer from spectral bias when using ReLU activations, limiting their ability to efficiently model high-frequency continuous signals (e.g., fine-grained images and audio). To address this, we propose the Learnable Sine Activation Function (STAF), a differentiable, end-to-end trainable sinusoidal activation that enables adaptive frequency modulation. STAF overcomes ReLU’s inherent spectral bias and the capacity-convergence trade-off. Its core components include: (i) a differentiable sine-based activation design, (ii) a spectral-adaptive modulation mechanism, and (iii) a joint optimization framework. Evaluated across diverse signal reconstruction tasks—including image and audio synthesis—STAF achieves consistent improvements: average PSNR gains of +1.2 dB, 37% faster convergence, and superior reconstruction fidelity compared to state-of-the-art methods.

Technology Category

• Machine Learning: Active Learning
• Natural Language Processing: Learning & Optimization for NLP
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Implicit Neural Representations (INRs) have emerged as a powerful framework for modeling continuous signals. The spectral bias of ReLU-based networks is a well-established limitation, restricting their ability to capture fine-grained details in target signals. While previous works have attempted to mitigate this issue through frequency-based encodings or architectural modifications, these approaches often introduce additional complexity and do not fully address the underlying challenge of learning high-frequency components efficiently. We introduce Sinusoidal Trainable Activation Functions (STAF), designed to directly tackle this limitation by enabling networks to adaptively learn and represent complex signals with higher precision and efficiency. STAF inherently modulates its frequency components, allowing for self-adaptive spectral learning. This capability significantly improves convergence speed and expressivity, making STAF highly effective for both signal representations and inverse problems. Through extensive evaluations, we demonstrate that STAF outperforms state-of-the-art (SOTA) methods in accuracy and reconstruction fidelity with superior Peak Signal-to-Noise Ratio (PSNR). These results establish STAF as a robust solution for overcoming spectral bias and the capacity-convergence gap, making it valuable for computer graphics and related fields. Our codebase is publicly accessible on the https://github.com/AlirezaMorsali/STAF.

Problem

• Neural Networks
• Complex Signal Processing
• ReLU Limitations

Innovation

• STAF activation function
• self-regulating frequency
• complex signal processing