Source-Learned Reliance for Selective Test-Time Adaptation of Multimodal Time Series

2026-10-05

AI Summary

This study addresses the limited adaptive reliability of multimodal wearable systems under missing sensors, which arises from misleading cross-modal consistency and high evaluation overhead. To this end, we propose CARAT, a framework that amortizes modality dependency estimation into the source training phase for the first time. By employing asymmetric modal dropout curriculum learning to precompute frozen dependency proxies, CARAT decouples model dependencies from runtime detection, thereby eliminating the computational cost of evaluating candidate subsets during deployment while enabling efficient and robust adaptation via selective representation fusion. Experimental results demonstrate that CARAT outperforms the strongest baseline, EATA, by 1.58 points in macro F1-score across four datasets, while reducing computational cost by 9.49% and parameter updates by 47.82%, effectively balancing robustness with efficiency.

Abstract

Multimodal wearable systems must remain reliable when sensor streams become noisy or unavailable. Existing multimodal test-time adaptation (TTA) methods often assess reliability online, but cross-modal agreement can be misleading when sensors measure different physical processes, and evaluating alternative modality configurations adds inference cost. We propose CARAT, which decouples model reliance from runtime corruption detection to guide omission or attenuation, amortizing reliance estimation through source training. An asymmetric modality-dropout curriculum prepares a missingness-resilient backbone for omission and derives a frozen, backbone-specific reliance proxy from windowed input-projection gradient norms. At deployment, a lightweight one-class detector flags suspect streams, and the proxy guides a joint choice between replacing the suspect set with the backbone's trained missingness symbol and attenuating its representations before fusion, without candidate-subset evaluation. Across four wearable datasets, five corruption types, three backbones, and eight TTA baselines, CARAT achieves the highest overall macro-F1 and best mean rank (2.42), exceeding EATA, the strongest baseline, by 1.58 F1 points across 12 equally weighted dataset-backbone settings. Across five profiled configurations, CARAT uses 9.49% fewer GFLOPs and updates 47.82% fewer parameters than EATA. A pattern also emerges across sensing regimes: multimodal TTA methods such as PTA are competitive on IMU-dominated homogeneous datasets, whereas unimodal TTA methods like TENT and EATA match or exceed it on heterogeneous datasets. These results position CARAT as a practical default to wearable TTA, offering competitive robustness with modest computational requirements and benefits that vary across backbones and dataset regimes.

Problem

• Multimodal time series
• Test-time adaptation
• Wearable systems
• Sensor corruption
• Reliability assessment

Innovation

• Test-Time Adaptation
• Multimodal Time Series
• Modality Dropout
• Reliance Proxy
• Wearable Systems