ERank in Latent Space as an Image-Complexity and Richness Measure

2026-07-21

AI Summary

This work addresses the absence of label-free, single-image metrics for visual richness by proposing the first unsupervised measure based on the effective rank (ERank) of channel-wise covariance matrices of deep features. The method leverages a frozen pretrained encoder to extract high-level image representations and quantifies visual complexity by computing the ERank of their channel covariance matrix, which reflects the number of decorrelated activation directions in the feature space. The proposed metric exhibits strong correlation with human-annotated complexity (r = 0.72) and effectively guides data selection for downstream tasks such as super-resolution and optical character recognition, yielding significant performance improvements.

Abstract

We propose the effective rank (ERank) of the channel covariance of an image's deep feature map as a per-sample, label-free measure of visual richness, computed from a single forward pass through a frozen pretrained encoder. ERank counts how many decorrelated channel directions an image activates, and we characterize its properties, including its behavior under noise. Empirically, ERank orders images from plain to visually rich, correlates with codec bitrate, sharpness, and edge density, and correlates with human complexity annotations on IC9600 with $r = 0.72$. As a data-selection criterion, removing low-ERank samples improves super-resolution and removing high-ERank samples improves OCR, in both pretraining and finetuning, while selection does not help classification, segmentation, or denoising. ERank is thus a cheap richness signal, useful exactly when task difficulty is governed by input richness.

Problem

• image complexity
• visual richness
• effective rank
• deep features
• data selection

Innovation

• effective rank
• image complexity
• latent space
• data selection
• visual richness