Self-supervised Transformation Learning for Equivariant Representations

2025-01-15

AI Summary

Existing equivariant learning methods struggle to model image invariance and equivariance under unseen transformations, hindering tasks requiring transformation-sensitive information—such as object localization and fine-grained classification. To address this, we propose Self-supervised Transformation Learning (STL), a novel framework that estimates transformation representations implicitly from image pairs without manual transformation labels, thereby decoupling transformation dependence and supporting complex augmentations (e.g., AugMix). STL synergistically integrates equivariant representation modeling with contrastive learning to jointly optimize both invariance and equivariance. Evaluated on 11 benchmarks, STL surpasses state-of-the-art methods on 7 tasks, with particularly notable gains in detection performance. It is compatible with mainstream backbone architectures and its implementation is publicly available.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Unsupervised representation learning has significantly advanced various machine learning tasks. In the computer vision domain, state-of-the-art approaches utilize transformations like random crop and color jitter to achieve invariant representations, embedding semantically the same inputs despite transformations. However, this can degrade performance in tasks requiring precise features, such as localization or flower classification. To address this, recent research incorporates equivariant representation learning, which captures transformation-sensitive information. However, current methods depend on transformation labels and thus struggle with interdependency and complex transformations. We propose Self-supervised Transformation Learning (STL), replacing transformation labels with transformation representations derived from image pairs. The proposed method ensures transformation representation is image-invariant and learns corresponding equivariant transformations, enhancing performance without increased batch complexity. We demonstrate the approach's effectiveness across diverse classification and detection tasks, outperforming existing methods in 7 out of 11 benchmarks and excelling in detection. By integrating complex transformations like AugMix, unusable by prior equivariant methods, this approach enhances performance across tasks, underscoring its adaptability and resilience. Additionally, its compatibility with various base models highlights its flexibility and broad applicability. The code is available at https://github.com/jaemyung-u/stl.

Problem

• Equivariant Learning
• Complex Image Transformations
• Invariant Features

Innovation

• STL
• Self-supervised Learning
• Image Transformation Invariance