🤖 AI Summary
Existing neural networks are constrained by hierarchical tree-like architectures, which preclude direct communication among sibling nodes and prohibit backward signal propagation to higher-level modules—resulting in weak inter-module collaboration and inefficient representation learning. To address these limitations, we propose the Synchronous Graph Neural Architecture (SGNA), organizing neural units into a modular, dynamically collaborative graph structure that enables arbitrary node-to-node communication and cross-layer signal transmission. Our key contributions are threefold: (1) introducing the first modular graph-structured paradigm for neural architecture design; (2) developing a systematic regularization framework to enforce module independence and load balancing; and (3) generalizing neural architecture search (NAS) to the space of directed acyclic graphs (DAGs). Extensive multi-task experiments demonstrate that SGNA significantly outperforms deep stacked baselines under parameter constraints, achieving superior collaborative representation capability and more comprehensive coverage of the search space.
📝 Abstract
This work proposes a new perspective on the structure of Neural Networks (NNs). Traditional Neural Networks are typically tree-like structures for convenience, which can be predefined or learned by NAS methods. However, such a structure can not facilitate communications between nodes at the same level or signal transmissions to previous levels. These defects prevent effective collaboration, restricting the capabilities of neural networks. It is well-acknowledged that the biological neural system contains billions of neural units. Their connections are far more complicated than the current NN structure. To enhance the representational ability of neural networks, existing works try to increase the depth of the neural network and introduce more parameters. However, they all have limitations with constrained parameters. In this work, we introduce a synchronous graph-based structure to establish a novel way of organizing the neural units: the Neural Modules. This framework allows any nodes to communicate with each other and encourages neural units to work collectively, demonstrating a departure from the conventional constrained paradigm. Such a structure also provides more candidates for the NAS methods. Furthermore, we also propose an elegant regularization method to organize neural units into multiple independent, balanced neural modules systematically. This would be convenient for handling these neural modules in parallel. Compared to traditional NNs, our method unlocks the potential of NNs from tree-like structures to general graphs and makes NNs be optimized in an almost complete set. Our approach proves adaptable to diverse tasks, offering compatibility across various scenarios. Quantitative experimental results substantiate the potential of our structure, indicating the improvement of NNs.