Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks

2026-07-23

AI Summary

This work addresses the challenge of unbiased interpretation of internal representations in deep neural networks, which is hindered by scale symmetries and structural biases. To this end, it introduces the first data-agnostic, hyperparameter-free unified mathematical framework: neurons are modeled as rank-1 Hilbert–Schmidt operators in a Hilbert space, enabling both pruning and neuron merging to be formulated as low-rank subspace projections. The framework naturally extends to the removal of macroscopic multi-layer modules. By providing an unbiased mechanism for cross-layer and cross-architecture representation dissection and compression decisions, the method achieves substantial performance retention in model compression and fine-tuning tasks, demonstrating its potential as an efficient, theoretically grounded, and structurally parsimonious approach to network compression.

Abstract

Deep neural networks encode complex representations, but deconstructing this internal knowledge remains a challenge. Given the link between learning and compression, network compression offers a promising lens to analyze this knowledge. However, standard compression heuristics often suffer from scale symmetries and architectural biases. To resolve these, we introduce Hilbert Operator for Progressive Encoding (HOPE), a mathematical framework to gradually deconstruct the representations in trained network weights. HOPE shifts network compression from the discrete domain into a Hilbert space of continuous functions. By modeling individual neurons as rank-1 Hilbert-Schmidt operators, HOPE unifies pruning and neuron merging as low-rank subspace projection. Extending this formulation, HOPE introduces macro block eviction to encompass multi-layer structures like entire residual pathways under the same unified metric. This unified approach enables unbiased architectural decisions across layers with different types and sizes. HOPE is a data-free and hyperparameter-free framework. We present proof-of-concept experiments in model compression and fine-tuning to highlight the practical potential of our theory.

Problem

• deep networks
• learned representations
• network compression
• scale symmetries
• architectural biases

Innovation

• Hilbert-Schmidt operator
• progressive encoding
• network compression
• low-rank subspace projection
• architecture-agnostic