FAN: Fourier Analysis Networks

2024-10-03

arXiv.org

AI Summary

Existing general-purpose neural networks (e.g., MLPs, Transformers) lack built-in inductive bias for periodic patterns, limiting their out-of-distribution (OOD) generalization and performance on periodic tasks such as time-series forecasting and symbolic reasoning. To address this, we propose Fourier Adaptive Networks (FAN), the first architecture to deeply integrate Fourier principles into generic neural network design. FAN introduces parameterized frequency-domain activations, learnable periodic embeddings, and a lightweight spectral transformation module—collectively enabling native support for periodic priors. Crucially, FAN remains task-agnostic and serves as a drop-in replacement for MLPs. Extensive experiments demonstrate that FAN significantly outperforms both MLPs and Transformers across diverse benchmarks—including symbolic formula representation, time-series forecasting, language modeling, and image recognition—while using fewer parameters and FLOPs. Moreover, FAN achieves substantial improvements in OOD generalization, validating its enhanced capacity to capture and extrapolate periodic structure.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Despite the remarkable successes of general-purpose neural networks, such as MLPs and Transformers, we find that they exhibit notable shortcomings in modeling and reasoning about periodic phenomena, achieving only marginal performance within the training domain and failing to generalize effectively to out-of-domain (OOD) scenarios. Periodicity is ubiquitous throughout nature and science. Therefore, neural networks should be equipped with the essential ability to model and handle periodicity. In this work, we propose FAN, a novel general-purpose neural network that offers broad applicability similar to MLP while effectively addressing periodicity modeling challenges. Periodicity is naturally integrated into FAN's structure and computational processes by introducing the Fourier Principle. Unlike existing Fourier-based networks, which possess particular periodicity modeling abilities but are typically designed for specific tasks, our approach maintains the general-purpose modeling capability. Therefore, FAN can seamlessly replace MLP in various model architectures with fewer parameters and FLOPs. Through extensive experiments, we demonstrate the superiority of FAN in periodicity modeling tasks and the effectiveness and generalizability of FAN across a range of real-world tasks, e.g., symbolic formula representation, time series forecasting, language modeling, and image recognition.

Problem

• Periodic Phenomena
• Neural Networks
• Adaptability

Innovation

• Fourier Analysis Network
• Periodic Phenomena Processing
• Universal Applicability