Ensemble and Mixture-of-Experts DeepONets For Operator Learning

2024-05-20

arXiv.org

AI Summary

To address the limited expressivity and generalization capability of DeepONet in learning operators for two- and three-dimensional partial differential equations (PDEs), this paper proposes PoU-MoE DeepONet—a novel multi-trunk ensemble architecture that integrates partition-of-unity (PoU)-based sparse modeling with a spatial mixture-of-experts (MoE) mechanism, systematically augmented by basis enhancement and POD-guided spatial localization. We establish its universal approximation property theoretically. On multiple 2D/3D PDE operator learning benchmarks, PoU-MoE DeepONet achieves 2–4× lower relative ℓ₂ error compared to standard DeepONet and POD-DeepONet, significantly improving accuracy, robustness, and spatially localized representation. The framework offers a scalable and interpretable paradigm for learning complex physical operators.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We present a novel deep operator network (DeepONet) architecture for operator learning, the ensemble DeepONet, that allows for enriching the trunk network of a single DeepONet with multiple distinct trunk networks. This trunk enrichment allows for greater expressivity and generalization capabilities over a range of operator learning problems. We also present a spatial mixture-of-experts (MoE) DeepONet trunk network architecture that utilizes a partition-of-unity (PoU) approximation to promote spatial locality and model sparsity in the operator learning problem. We first prove that both the ensemble and PoU-MoE DeepONets are universal approximators. We then demonstrate that ensemble DeepONets containing a trunk ensemble of a standard trunk, the PoU-MoE trunk, and/or a proper orthogonal decomposition (POD) trunk can achieve 2-4x lower relative $ell_2$ errors than standard DeepONets and POD-DeepONets on both standard and challenging new operator learning problems involving partial differential equations (PDEs) in two and three dimensions. Our new PoU-MoE formulation provides a natural way to incorporate spatial locality and model sparsity into any neural network architecture, while our new ensemble DeepONet provides a powerful and general framework for incorporating basis enrichment in scientific machine learning architectures for operator learning.

Problem

• Enhance operator learning with ensemble DeepONet architecture
• Introduce spatial mixture-of-experts for model sparsity and locality
• Achieve lower errors in PDE-related operator learning problems

Innovation

• Ensemble DeepONet enhances trunk network expressivity.
• Spatial MoE DeepONet promotes locality and sparsity.
• Combines multiple trunk types for improved error rates.