Cluster Validation Indices as Self-Supervised Objectives for Text Representation Learning

2026-10-04

AI Summary

This study addresses the high computational overhead and reliance on negative samples and data augmentation in contrastive learning for self-supervised fine-tuning by proposing SilK. This method is the first to replace external negative samples with an internal clustering quality metric, specifically the silhouette coefficient, reformulating the optimization objective as centroid-level contrast. This enables lightweight training using single views without data augmentation. Experiments based on BERT-base with linear probing demonstrate that SilK achieves downstream task performance comparable to the strongest baselines while accelerating training speed by 1.46× and reducing GPU memory consumption by 45.4%. These results establish SilK as a novel paradigm for efficient self-supervised representation learning.

Abstract

Self-supervised fine-tuning refines the embedding space of a pretrained language encoder without labels. However, the commonly used approaches are computationally expensive. Specifically, contrastive learning-based methods need multiview data and in-batch negative examples, while negative-free approaches require auxiliary graphs/networks. An interesting question arises: can self-supervised fine-tuning be done without relying on either additional negatives or graph data? To answer this question, we introduce SilK (Silhouette-guided K-means), which trains on a Cluster Validation Index, an internal measure of cluster quality without using labels. SilK clusters the corpus and then regresses a simplified silhouette toward a target value. Each document is compared only against the k cluster centroids, never against other documents, so the method needs no augmentation, no negative pairs and one view per document. On BERT-base, SilK trains 1.46x faster per epoch than the fastest baseline we evaluate and uses 45.4% less peak GPU memory than the leanest one. Under frozen-encoder linear probing, SilK stays competitive with the best baselines on three downstream tasks.

Problem

• self-supervised fine-tuning
• text representation learning
• contrastive learning
• cluster validation indices
• computational efficiency

Innovation

• Self-supervised learning
• Cluster Validation Index
• Text representation learning
• Silhouette coefficient
• Contrastive-free fine-tuning