Adversarially Robust Abductive Fusion of Pre-trained Transformer-based Perception Models

2026-08-04

AI Summary

Pretrained perceptual models suffer significant performance degradation under distribution shifts, and conventional fusion methods are vulnerable to co-failure and adversarial attacks. This work proposes the Label Vector Pooling (LVP) framework, which automatically learns error-detection rules by modeling the geometric relationships among model embeddings and formulates multi-model fusion as an abductive inference problem under consistency constraints. The solution combines integer programming with polynomial-time heuristics. LVP achieves, for the first time, domain-agnostic metacognitive error detection by unifying geometric reasoning with neuro-symbolic logic, offering both interpretability and adversarial robustness. Evaluated on 15 aerial image datasets under weather-induced distribution shifts, LVP attains a clean-data F1 score only 0.005 lower than optimal majority voting; under 90% label-flip attacks, it achieves an average F1 of 0.42—outperforming majority voting (0.35)—and consistently yields the highest performance across all test sets when the flip rate exceeds 0.4.

Abstract

Deploying pre-trained perception models in novel environments degrades their accuracy under distributional shift, and assembling them alone does not recover it: combiners such as majority voting trade recall for precision and are brittle to coordinated failures. Prior metacognitive methods learn logical rules that flag a model's errors, but rely on hand-authored domain-knowledge cues (object-size priors, segmentation masks) that do not transfer to genuinely novel scenes. We show that this metacognitive layer can be learned without any domain knowledge by exploiting vector-space geometry: per-model Label Vector Pools (LVP), built from each model's own training embeddings, yield error-detection rules from the geometry of detections relative to training-determined prototypes, reaching parity with domain-knowledge rules to within $0.002$ every F1 on test set. Because the approach remains neurosymbolic, these geometric rules share a single logical framework and can still be complemented by domain knowledge when available. We frame the fusion of multiple imperfect ViT-based detectors as a consistency-based abduction problem solved at test time by an exact Integer Program (IP) and a polynomial-time heuristic. On an aerial-imagery benchmark of 15 weather-shifted test sets and six ViT detectors, our domain-knowledge-free layer matches the strongest majority-vote variant on clean data (within $0.005$ F1) and, unlike every majority-vote baseline, retains its performance under a coordinated label-flipping attack: at a $90\%$ flip rate it averages $0.42$ F1 versus $0.35$ for MV-Plurality (a $22\%$ relative gain) and attains the highest F1 on \emph{every} test set once the flip rate exceeds $0.4$

Problem

• distributional shift
• model fusion
• adversarial robustness
• domain generalization
• perception models

Innovation

• adversarially robust fusion
• label vector pools
• neurosymbolic abduction
• distributional shift
• vision transformer ensemble