TASAR: Transfer-based Attack on Skeletal Action Recognition

2024-09-04

AI Summary

Existing adversarial attacks for skeleton-based action recognition suffer from poor cross-model transferability and neglect temporal continuity of human motion, limiting their effectiveness against real-world recognition systems (e.g., autonomous driving). This work identifies sharpness of the loss landscape as a key factor hindering transferability. We propose the first transfer-based attack framework that requires no proxy model retraining: it employs dual Bayesian posterior smoothing for gradient-robust optimization and—novelly—integrates motion dynamics modeling into adversarial perturbation generation to disrupt models’ reliance on spatiotemporal consistency. Extensive evaluations across seven models and three benchmark datasets demonstrate significant improvements over ten state-of-the-art baselines. Furthermore, we introduce S-HAR, the first open-source robustness evaluation benchmark for skeleton-based HAR, and release all code publicly.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Adversarial Search

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Skeletal sequences, as well-structured representations of human behaviors, play a vital role in Human Activity Recognition (HAR). The transferability of adversarial skeletal sequences enables attacks in real-world HAR scenarios, such as autonomous driving, intelligent surveillance, and human-computer interactions. However, most existing skeleton-based HAR (S-HAR) attacks are primarily designed for white-box scenarios and exhibit weak adversarial transferability. Therefore, they cannot be considered true transfer-based S-HAR attacks. More importantly, the reason for this failure remains unclear. In this paper, we study this phenomenon through the lens of loss surface, and find that its sharpness contributes to the weak transferability in S-HAR. Inspired by this observation, we assume and empirically validate that smoothening the rugged loss landscape could potentially improve adversarial transferability in S-HAR. To this end, we propose the first extbf{T}ransfer-based extbf{A}ttack on extbf{S}keletal extbf{A}ction extbf{R}ecognition, TASAR. TASAR explores the smoothed model posterior without requiring surrogate re-training, which is achieved by a new post-train Dual Bayesian optimization strategy. Furthermore, unlike previous transfer-based attacks that treat each frame independently and overlook temporal coherence within sequences, TASAR incorporates motion dynamics into the Bayesian attack gradient, effectively disrupting the spatial-temporal coherence of S-HARs. To exhaustively evaluate the effectiveness of existing methods and our method, we build the first large-scale robust S-HAR benchmark, comprising 7 S-HAR models, 10 attack methods, 3 S-HAR datasets and 2 defense methods. Extensive results demonstrate the superiority of TASAR. Our benchmark enables easy comparisons for future studies, with the code available in the supplementary material.

Problem

• skeleton action recognition
• adversarial attacks
• continuous motion

Innovation

• TASAR
• skeleton action recognition
• adversarial attack