Not All Forgetting Is Equal: Architecture-Dependent Retention Dynamics in Fine-Tuned Image Classifiers

2026-04-13

AI Summary

This study investigates how model architecture influences sample forgetting dynamics during fine-tuning. By tracking per-epoch prediction correctness on ResNet-18 and DeiT-Small, the authors fit Ebbinghaus-style exponential decay curves and analyze forgetting patterns using Spearman correlation, Jaccard overlap, and R² metrics. They find that CNNs and Vision Transformers (ViTs) forget markedly different samples, with ViTs exhibiting more structured forgetting. While sample-level forgetting appears stochastic, class-level forgetting demonstrates semantic consistency, and sample difficulty is shown not to be an intrinsic property. Further experiments reveal that spaced repetition sampling based on decay constants does not outperform random sampling, suggesting that static scheduling struggles to leverage unstable forgetting signals effectively. These findings highlight that architectural differences induce complementary forgetting behaviors, offering new insights for curriculum design and data pruning strategies.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Fine-tuning pretrained image classifiers is standard practice, yet which individual samples are forgotten during this process, and whether forgetting patterns are stable or architecture dependent, remains unclear. Understanding these dynamics has direct implications for curriculum design, data pruning, and ensemble construction. We track per-sample correctness at every epoch during fine-tuning of ResNet-18 and DeiT-Small on a retinal OCT dataset (7 classes, 56:1 imbalance) and CUB-200-2011 (200 bird species), fitting Ebbinghaus-style exponential decay curves to each sample's retention trace. Five findings emerge. First, the two architectures forget fundamentally different samples: Jaccard overlap of the top 10 percent most-forgotten is 0.34 on OCTDL and 0.15 on CUB-200. Second, ViT forgetting is more structured (mean $R^2 = 0.74$) than CNN forgetting ($R^2 = 0.52$). Third, per-sample forgetting is stochastic across random seeds (Spearman $ρ\approx 0.01$), challenging the assumption that sample difficulty is an intrinsic property. Fourth, class-level forgetting is consistent and semantically interpretable: visually similar species are forgotten most, distinctive ones least. Fifth, a sample's loss after head warmup predicts its long-term decay constant ($ρ= 0.30$ to $0.50$, $p < 10^{-45}$). These findings suggest that architectural diversity in ensembles provides complementary retention coverage, and that curriculum or pruning methods based on per-sample difficulty may not generalize across runs. A spaced repetition sampler built on these decay constants does not outperform random sampling, indicating that static scheduling cannot exploit unstable per-sample signals.

Problem

• forgetting
• fine-tuning
• architecture dependence
• sample retention
• image classification

Innovation

• forgetting dynamics
• architecture dependence
• fine-tuning
• sample retention
• vision transformers