ZeroLM: Data-Free Transformer Architecture Search for Language Models

2025-03-24

AI Summary

Existing zero-cost proxy methods exhibit poor performance in Transformer architecture ranking—often underperforming even simple parameter count baselines—and suffer from high search overhead, overfitting susceptibility, and modeling complexity. This paper proposes a training-free, gradient-free zero-cost method for Transformer architecture search. We introduce the first weight-statistics-based proxy metric and, for the first time, decouple Transformers into functional submodules, dynamically weighting their capacity contributions to overcome ranking performance bottlenecks. On the FlexiBERT benchmark, our method achieves a Spearman correlation of 0.76 and Kendall’s tau of 0.53 with ground-truth validation accuracy, while incurring near-zero search cost. It demonstrates strong cross-task robustness and significantly outperforms existing zero-cost approaches.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Neural architecture search (NAS) provides a systematic framework for automating the design of neural network architectures, yet its widespread adoption is hindered by prohibitive computational requirements. Existing zero-cost proxy methods, while reducing search overhead, demonstrate inadequate performance in architecture ranking tasks, particularly for Transformer-based models where they often underperform simple parameter counting metrics. Current automated proxy discovery approaches suffer from extended search times, susceptibility to data overfitting, and structural complexity. This paper introduces a novel zero-cost proxy methodology that quantifies model capacity through efficient weight statistics computation while decomposing Transformer architectures into functionally distinct sub-modules, thereby optimizing the balance of their contributions to overall performance. Our comprehensive evaluation demonstrates the superiority of this approach, achieving a Spearman's rho of 0.76 and Kendall's tau of 0.53 on the FlexiBERT benchmark. The proposed method exhibits exceptional computational efficiency while maintaining robust performance across diverse NAS benchmark tasks, offering a practical solution for large-scale architecture search.

Problem

• Automates neural architecture design with low computational cost
• Improves ranking accuracy for Transformer-based model architectures
• Balances sub-module contributions to enhance overall model performance

Innovation

• Zero-cost proxy for efficient architecture ranking
• Decomposes Transformers into functional sub-modules
• Uses weight statistics to quantify model capacity