Technical note on: Zero-Training Feature-Space Alignment via Information Geometry

2026-09-29

AI Summary

This study addresses the degradation of robustness in deep vision models under distribution shifts by proposing a training-free Fisher geometric alignment method. Grounded in information geometry, the approach estimates the Fisher information matrix and introduces natural gradient-inspired preconditioning to correct geometric distortions in the feature space via a closed-form linear transformation. As a deterministic test-time adaptation scheme, it requires no parameter updates, iterative computations, or hyperparameter tuning. Experimental results demonstrate that the proposed method consistently improves performance across multiple benchmarks, emerging as the only approach that degrades none of the evaluated model families. Furthermore, the observed performance gains exhibit a significant positive correlation with the degree of geometric distortion.

Abstract

Deep vision models often degrade under distribution shift. Test-time adaptation can improve robustness but typically requires iterative optimization, hyperparameter tuning, and multiple forward-backward passes. We propose Zero-Training Fisher Geometry Alignment (ZFGA), a closed-form method that improves robustness under covariate shift without modifying model parameters. ZFGA is based on the observation that distribution shifts distort feature-space geometry. It estimates the Fisher information matrix of the predictive distribution with respect to feature embeddings and applies a linear transformation that aligns test-feature Fisher geometry with a reference geometry computed from clean data. This provides a natural-gradient-inspired preconditioning step in feature space. We evaluate ZFGA on CIFAR-10-C and ImageNet-C using ResNet-50, DINO ViT-S/16, and CLIP ViT-B/32. ZFGA consistently improves over zero-shot inference across all three models, although it is not the strongest method for every model. Covariance whitening performs better on ResNet-50, while Fisher whitening is statistically indistinguishable from ZFGA on CLIP. Across six training-free and gradient-based alternatives (covariance whitening, Fisher whitening, TENT, T3A, LAME, and AdaNPC), ZFGA is the only method that does not substantially harm any of the three model families. The Fisher geometry distortion is also positively correlated with ZFGA gain (Pearson r = 0.366, p = 0.017), providing preliminary evidence that geometric misalignment contributes to robustness degradation. ZFGA requires only forward passes and matrix operations at inference time, offering a lightweight and deterministic alternative to optimization-based test-time adaptation.

Problem

• distribution shift
• test-time adaptation
• robustness
• feature-space alignment
• training-free

Innovation

• Zero-Training Adaptation
• Fisher Information Matrix
• Feature-Space Alignment
• Information Geometry
• Test-Time Robustness