Effective Does Not Mean Useful: Conditional Functional Substitutability for Redundancy and Scaling in Transformers

2026-09-30

AI Summary

This study addresses the inability of conventional methods to directly characterize dynamic functional redundancy in Transformers by proposing the Conditional Functional Substitutability (CFS) framework. Moving beyond indirect proxies based on importance or similarity, CFS defines redundancy as an input-conditioned dynamic relationship, directly capturing functional substitution through intermediate states that induce similar downstream responses. Combined with cross-modal analysis and controlled scaling experiments, this approach reveals systematic functional reorganization and scaling properties. Results demonstrate that performance gains do not necessarily increase monotonically with substitutability; rather, independent functional structures more effectively enhance model performance. Furthermore, the proposed framework offers a functional explanation for diminishing returns and achieves superior computation-performance trade-offs compared to traditional methods.

Abstract

Modern neural networks scale predictably, yet the mechanisms behind these regularities remain unclear. Neural redundancy is typically characterized by component importance or representational similarity, both indirect proxies. We view redundancy as an input-conditioned, dynamic relation: intermediate computational states are functionally redundant when they induce similar downstream responses. We introduce Conditional Functional Substitutability (CFS) to directly characterize such functional substitution. CFS exposes functional relations and reduction potential missed by conventional importance- and similarity-based measures. Across modalities and Transformer families, CFS reveals systematic functional reorganization with scale. Controlled scaling further shows that performance gains need not track growth in substitutability, while fixed-capacity models with more independent functional structure perform better, providing a functional account of diminishing returns. Predicted CFS further enables dynamic computation with a better performance--computation trade-off than importance-based component selection, suggesting new directions for redundancy-aware computation and more efficient model scaling.

Problem

• neural redundancy
• functional substitutability
• Transformer scaling
• diminishing returns
• dynamic computation

Innovation

• Conditional Functional Substitutability
• Neural Redundancy
• Transformers
• Dynamic Computation
• Model Scaling