Bridging the sim2real gap in the table tennis robot with a transformer-based ball states predictor

2026-06-09

AI Summary

This work addresses the sim-to-real transfer challenge faced by table tennis robots in dynamic environments, where inaccurate ball state prediction hinders effective policy deployment. To overcome this, the authors propose a Transformer-based ball state prediction framework that leverages attention mechanisms to model long-term temporal dependencies from historical observations, eliminating the need for explicit ballistic models and instead relying on large-scale real-world data for training. A novel, plug-and-play SPAD strategy is introduced at deployment time, enabling seamless transfer of control policies learned in simulation to real-world systems without requiring retraining. This approach substantially narrows the sim-to-real gap, achieves high-accuracy long-horizon trajectory prediction, and significantly enhances both control performance and policy transfer efficiency in real-world robotic table tennis tasks.

Abstract

Robotic table tennis is a representative benchmark for high-speed, closed-loop robotic control in dynamic environments, where accurate and fast prediction of ball states is critical for reliable planning and control. Physics-based approaches rely heavily on accurate parameter identification and precise initial state, while learning-based methods often struggle to capture long-range temporal dependencies and are typically trained on limited or simulated data. We propose a transformer-based framework for table tennis ball state prediction that leverages attention mechanisms to model long-range temporal correlations directly from historical observations, without relying on explicit flight or bounce models. To support robust learning and generalization, we collected a large-scale real-world dataset from players of varying skill levels and diverse ball cannon configurations. The combination of a high-capacity transformer architecture and extensive real-world data enables accurate long-horizon forecasting. Building on this capability, we introduce a plug-and-play sim-to-real transfer strategy, Swap Predictor at Deployment (SPAD), which replaces the physics-based simulator used during training with the proposed real-world-trained predictor at deployment, improving the sim-to-real transferability of the policy without requiring retraining. We demonstrate that this simple substitution effectively narrows the sim-to-real gap while preserving the efficiency and scalability of simulation-based training.

Problem

• sim2real gap
• table tennis robot
• ball state prediction
• high-speed robotic control
• dynamic environments

Innovation

• transformer-based prediction
• sim2real transfer
• long-range temporal modeling
• SPAD
• real-world dataset