LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations

2026-09-23

AI Summary

本文提出LAYERSCOPE框架，通过几何度量无标签地评估视频和多模态模型各层学习表示，解决了需要大量标记数据和计算的问题。

Abstract

We propose LAYERSCOPE, a label-free, layerwise framework that aims to characterize a model's learned representations in video and multimodal settings. Evaluating downstream performance using representations from final or intermediate layers typically requires large amounts of labeled data, repeated task-specific evaluations, and substantial computation. To address these limitations, LAYERSCOPE uses local, global, distributional, and correspondence-based geometric metrics to compare layerwise representation structure within and across models without requiring task-specific labels. We evaluate seven architecturally diverse models across video and multimodal classification, clustering, and text-to-video retrieval tasks from MVEB/MVEB+. We find that intermediate-layer representations can outperform final-layer and model-default outputs. We also find that no single geometric metric consistently predicts downstream performance, but note that distinct layerwise geometric signatures emerge across model families. LID shows task-dependent relationships with performance, while RankMe provides the strongest measure for classification and clustering, but is not a universal layer selector. We also find that pairing-aware metrics explain retrieval better than distributional distances alone. LAYERSCOPE therefore offers a framework for comparing representations across models and layers, enabling a more systematic evaluation in video and multimodal settings.

Problem

• video
• multimodal
• learned representations
• labeled data
• computation

Innovation

• label-free
• layerwise framework
• geometric metrics
• multimodal representations
• video representations