perceptual reconstruction loss

This work identifies that vision encoders (e.g., CLIP) implicitly encode imperceptible device- and algorithm-specific artifacts—such as camera model, compression parameters, and post-processing pipelines—introduced during image acquisition and processing. It systematically evaluates how these non-semantic signals interfere with or enhance downstream semantic predictions. Using feature interpretability analysis, linear probing, controlled ablation experiments, and distributional correlation modeling, the study quantitatively demonstrates: (1) acquisition and processing parameters are highly recoverable from CLIP visual representations (mean accuracy >92%); (2) such artifacts significantly degrade classification robustness under distribution shifts, inducing up to 15.3% error fluctuation; and (3) their statistical correlation with semantic labels modulates prediction confidence—either positively or negatively. This is the first systematic investigation revealing “implicit metadata contamination” in vision representations—a previously overlooked source of spurious correlation. To foster reproducibility, the authors release all code and benchmark datasets.

This work addresses the challenge of enhancing interpretability in deep learning by introducing the **Loss Kernel**—a novel geometric probe grounded in the covariance structure of sample-wise losses. Methodologically, it defines an intrinsic similarity metric among data points by modeling how individual sample losses co-vary under infinitesimal parameter perturbations. This approach uniquely bridges statistical stability in loss space with kernel-based representation learning. Theoretically, the Loss Kernel is proven to separate inputs according to semantic task structure; empirically, it uncovers hierarchical semantic organization within deep networks. Visualizations on Inception-v1 reveal interpretable feature-level relationships; validation on synthetic multi-task datasets confirms its fidelity; and on ImageNet, its implicit clustering aligns closely with the WordNet semantic hierarchy. Collectively, the Loss Kernel provides a principled, geometric, and generalizable framework for interpreting deep neural representations—offering both theoretical guarantees and practical insights into latent semantic structure.

This work addresses the lack of clear information-theoretic guidance in existing perceptual lossy source coding, which has led to overreliance on opaque deep learning models. Building upon rate–distortion–perception theory, the study reformulates information-theoretic limits from mere performance benchmarks into constructive design principles. Through an intuitive unit-circle teaching model, it elucidates core mechanisms—including codec architecture, trade-off relationships, and universal representations—and clarifies the role of common randomness in unifying single-shot and asymptotic settings. The proposed framework bridges perceptual coding with classical lossy compression, offering a theoretically grounded foundation for interpretable and implementable system design along with principled optimization pathways.

Traditional image compression evaluation relies on distortion metrics such as MSE, which exhibit significant misalignment with human perceptual judgment. To address this, we propose VLIC—the first perception-aligned compression framework leveraging diffusion models and frozen vision-language models (VLMs, e.g., CLIP or LLaVA) as zero-shot preference discriminators. Crucially, VLIC bypasses fine-tuning or distillation; instead, it employs VLMs to perform binary Alternative Forced Choice (AFC) comparisons on compressed image pairs, generating preference-based reward signals to guide diffusion model post-training. This paradigm establishes the first native, parameter-free VLM-driven perceptual guidance for compression. Extensive experiments demonstrate that VLIC achieves state-of-the-art performance across perceptual metrics—including LPIPS and DISTS—as well as in large-scale user studies, significantly outperforming both classical and learned compression methods.

This work identifies a critical asymmetry between image quality assessment (IQA) and perceptual optimization: high-fidelity IQA metrics (e.g., LPIPS) perform poorly in perceptual optimization tasks—especially under adversarial training—revealing a structural mismatch between evaluation and optimization objectives. Method: We systematically compare convolutional, patch-based, and Transformer-based discriminators in both generative optimization and IQA, analyzing their architectural impact on detail reconstruction, artifact suppression, and representation transferability. Contribution/Results: We find that patch-based convolutional discriminators achieve optimal perceptual optimization performance, yet their learned representations yield limited gains when transferred to IQA model backbones. This demonstrates that discriminator architecture critically governs optimization efficacy but does not produce universally transferable IQA representations. To our knowledge, this is the first empirical characterization of such an evaluation-optimization misalignment, establishing a new paradigm for discriminator design and IQA model development.

Existing image quality assessment methods primarily emphasize visual fidelity and struggle to capture the preservation of semantic content in low-level image processing. This work formally introduces the task of “semantic similarity” evaluation and proposes a structured semantic representation framework that decouples foreground and background entities. By integrating open-world category and relational modeling, the framework constructs image triplets representing semantic structures and introduces a Triplet Semantic Similarity (T3S) score for quantitative assessment. Experiments on COCO and SPA-Data demonstrate that T3S significantly outperforms existing fidelity-based metrics and semantic-level baselines, offering a more accurate characterization of progressive semantic changes under various image degradations.

This work reveals that the often-overlooked embedding norm in contrastive learning inherently encodes critical semantic information, such as semantic specificity. From the perspective of optimization dynamics, we theoretically demonstrate—for the first time—the mechanism by which embedding norms naturally capture semantic attributes during training under scale-invariant losses. We derive analytical relationships between the norm and established semantic metrics, including concept specificity, word frequency, and human uncertainty. Furthermore, we show that the norm serves as a calibration signal without requiring additional training, offering significant practical utility in retrieval and confidence calibration tasks.

This work addresses the degradation of visual perceptual quality caused by compression artifacts introduced by standard video codecs. To mitigate this issue, the authors propose a lightweight, codec-agnostic, semantic-aware preprocessing framework that integrates semantic embeddings from vision-language models with a differentiable codec proxy. The framework employs an efficient convolutional network to enable end-to-end training, selectively enhancing detail fidelity in perceptually salient regions without altering the existing encoding pipeline. Experimental results demonstrate that the proposed method significantly outperforms baseline approaches on high-resolution benchmarks, achieving consistent improvements in both MS-SSIM and VMAF metrics while effectively preserving texture details in visually critical areas.

This work addresses the overestimation of semantic shifts between large-scale natural image datasets in existing supervised classification methods, which often rely on non-semantic artifacts such as resolution rather than genuine semantic content. To rectify this, the authors propose the first unsupervised semantic clustering framework that leverages semantic features extracted from foundation vision models to directly assess inter-dataset semantic separability without requiring labels. Through carefully designed controlled experiments, they demonstrate that the high classification accuracy reported by conventional approaches primarily stems from non-semantic confounders. When applied to mainstream web-scale datasets, their method yields clustering performance near random chance, providing strong evidence that previously reported semantic discrepancies have been substantially exaggerated.

Post-training quantization of vision encoders (e.g., CLIP) often suffers severe accuracy degradation—particularly at 8-bit precision—due to prominent outliers in intermediate-layer activations. This paper identifies a key distinction: outlier distributions in vision encoders fundamentally differ from those in language models. To address this without fine-tuning, we propose RegCache, a lightweight, inference-only quantization optimization method. RegCache injects prefix tokens into critical intermediate layers and dynamically prunes redundant ones via token scheduling, selectively suppressing outlier activations at semantically sensitive positions. Crucially, it preserves the original model architecture and introduces no trainable parameters. Extensive experiments on both text-supervised and self-supervised vision encoders demonstrate that RegCache boosts 8-bit quantized model Top-1 accuracy by 4.2–6.8 percentage points on average, closely matching full-precision performance.