VirtualFencer: Generating Fencing Bouts based on Strategies Extracted from In-the-Wild Videos

2025-06-30

AI Summary

This paper introduces the first end-to-end framework for jointly learning 3D human motion and two-player adversarial strategies from unannotated, in-the-wild fencing videos. To address the absence of 3D pose annotations and tactical labels, the method integrates unsupervised 3D pose estimation, spatiotemporal graph-based strategy abstraction, and conditional sequence generation, enabling implicit extraction of motion dynamics and attack–defense logic via self-supervision. Key contributions include: (1) the first joint disentangled representation of action and strategy directly from unconstrained video; and (2) a controllable generative system producing diverse, tactically coherent sequences—supporting autonomous self-play, historical match replay, and real-time human–AI interaction. Experiments demonstrate that generated sequences achieve professional-level fidelity in motion realism, tactical plausibility, and response latency.

Technology Category

• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Fencing is a sport where athletes engage in diverse yet strategically logical motions. While most motions fall into a few high-level actions (e.g. step, lunge, parry), the execution can vary widely-fast vs. slow, large vs. small, offensive vs. defensive. Moreover, a fencer's actions are informed by a strategy that often comes in response to the opponent's behavior. This combination of motion diversity with underlying two-player strategy motivates the application of data-driven modeling to fencing. We present VirtualFencer, a system capable of extracting 3D fencing motion and strategy from in-the-wild video without supervision, and then using that extracted knowledge to generate realistic fencing behavior. We demonstrate the versatile capabilities of our system by having it (i) fence against itself (self-play), (ii) fence against a real fencer's motion from online video, and (iii) fence interactively against a professional fencer.

Problem

• Extracting 3D fencing motion and strategy from videos
• Generating realistic fencing behavior autonomously
• Simulating interactive bouts against real and professional fencers

Innovation

• Extracts 3D fencing motion from videos
• Models two-player strategy unsupervised
• Generates realistic interactive fencing behavior