On the Depth Scalability of Logic Gate Networks

2026-07-19

AI Summary

This work addresses the challenges of optimization collapse and topological constraints that hinder performance gains in deep traditional logic gate networks. To overcome these limitations, the authors propose Input-Anchored Logic Gate Networks (IALGN), which establish stable information pathways by directly connecting each layer’s logic gates to the original input. The approach integrates input-anchored topology, skip-biased initialization, straight-through gradient estimation, and Random-k anchor relaxation to enforce a strict path-depth hierarchy, ensuring effective utilization of input information even in very deep architectures. Using this framework, the authors successfully train logic gate networks exceeding 100 layers on MNIST, CIFAR-10, and CIFAR-100, achieving simultaneous improvements in depth and accuracy and significantly outperforming existing architectures.

Abstract

Logic Gate Networks (LGNs) implement computation through compositions of Boolean operations, yet unlike classical Boolean circuits, existing LGNs do not reliably benefit from increased depth. We identify two distinct causes: optimization collapse in deep relaxed LGNs and a topology-induced limitation that persists even when skip-biased initialization and straight-through estimation stabilize training. Thus, trainability alone is insufficient; deeper layers must also receive information that supports useful computation. We introduce Input-Anchored Logic Gate Networks (IALGNs), in which each gate combines an evolving hidden feature with a direct input anchor. This topology preserves a computational spine while conditioning every layer on the original input. We show that a depth-D path can depend on up to D+1 input bits and establish a strict path-wise depth hierarchy. Random-k anchor relaxation further improves anchor selection without relaxing the spine. Across MNIST, CIFAR-10, and CIFAR-100, IALGNs achieve consistent fixed-width depth--accuracy improvements beyond 100 layers, whereas alternative LGN topologies saturate or degrade. Layer-wise probes, topology ablations, and effective-depth analysis show that input anchoring produces progressively more informative representations and preserves longer computational paths. These results demonstrate that scalable depth in LGNs requires both stable optimization and an information-access pattern that supports input-conditioned refinement.

Problem

• Logic Gate Networks
• depth scalability
• optimization collapse
• information access
• computational paths

Innovation

• Input-Anchored Logic Gate Networks
• depth scalability
• computational spine
• Boolean circuits
• information-access pattern