Beyond Softmax: Dual-Branch Sigmoid Architecture for Accurate Class Activation Maps

2025-11-05

AI Summary

Existing Softmax-based Class Activation Mapping (CAM) methods suffer from two critical distortions: additive bias shifts importance scores, while sign collapse conflates excitatory and inhibitory features. To address these issues, we propose a dual-branch Sigmoid architecture that decouples classification and localization tasks: the original Softmax branch is frozen to preserve classification accuracy, while an auxiliary Sigmoid branch is fine-tuned under binary supervision to directly produce high-fidelity, signed (±) activation maps with calibrated magnitudes. This design overcomes inherent Softmax limitations—enabling improved interpretability and localization accuracy without compromising classification performance. Evaluated on CUB-200-2011 and ImageNet, our method achieves significant gains in localization accuracy (e.g., +5.2% Top-1 localization accuracy on CUB-200-2011) while maintaining original classification accuracy, and remains compatible with mainstream CAM variants.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Interpretability, Explainability, and Transparency
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Class Activation Mapping (CAM) and its extensions have become indispensable tools for visualizing the evidence behind deep network predictions. However, by relying on a final softmax classifier, these methods suffer from two fundamental distortions: additive logit shifts that arbitrarily bias importance scores, and sign collapse that conflates excitatory and inhibitory features. We propose a simple, architecture-agnostic dual-branch sigmoid head that decouples localization from classification. Given any pretrained model, we clone its classification head into a parallel branch ending in per-class sigmoid outputs, freeze the original softmax head, and fine-tune only the sigmoid branch with class-balanced binary supervision. At inference, softmax retains recognition accuracy, while class evidence maps are generated from the sigmoid branch -- preserving both magnitude and sign of feature contributions. Our method integrates seamlessly with most CAM variants and incurs negligible overhead. Extensive evaluations on fine-grained tasks (CUB-200-2011, Stanford Cars) and WSOL benchmarks (ImageNet-1K, OpenImages30K) show improved explanation fidelity and consistent Top-1 Localization gains -- without any drop in classification accuracy. Code is available at https://github.com/finallyupper/beyond-softmax.

Problem

• Addresses distortions in Class Activation Maps from softmax reliance
• Decouples localization from classification using dual-branch architecture
• Preserves feature contribution magnitude and sign for accurate visualization

Innovation

• Dual-branch sigmoid head decouples localization from classification
• Freezes original softmax head while fine-tuning sigmoid branch
• Generates class evidence maps preserving feature magnitude and sign