Mixture-of-experts for handwriting trajectory reconstruction from IMU sensors

2026-07-29

AI Summary

This work addresses the challenges of continuity and accuracy in reconstructing handwriting trajectories from digital pen IMU signals during both on-surface writing and in-air hovering phases. To this end, we propose a novel approach based on a Mixture-of-Experts (MoE) architecture, featuring dedicated expert models tailored specifically for each writing state—on-surface and hovering—and integrating contextual information to achieve high-fidelity trajectory reconstruction and precise prediction of the next pen-down location. To the best of our knowledge, this is the first study to introduce the MoE mechanism into handwriting trajectory reconstruction, demonstrating significant performance gains over existing methods, particularly in complex writing scenarios. Additionally, we release the first public benchmark dataset to facilitate further research in this domain.

Abstract

The use of digital pens for online handwriting trajectory reconstruction is a prevalent method for human-computer interaction. In this study, we focus on a digital pen equipped with sensors where we aim at reconstructing the online handwriting trajectory. This pen enables writing on any surface and preserving the digital trace of handwriting. This type of pen could be used as an aid to learning to write in classroom. In this paper, we propose a new approach learning to finely reconstruct the touching trajectories while precisely analyzing the hovering part in order to position the next touching trace correctly. This relies on a Mixture-Of-Experts (MOE) approach. The first expert is dedicated for the pencil touch, and is named touching expert model. The second one is dedicated for the hovering pen trajectory, and is named hovering expert model. We improve on the learning of each of these experts based on additional context or specific examples. In addition we introduce a novel public benchmark dataset, to enable future research and comparisons in the field of handwriting reconstruction. The results demonstrates a significant enhancement compared to its primary competitors.

Problem

• handwriting trajectory reconstruction
• IMU sensors
• digital pen
• touching trajectory
• hovering trajectory

Innovation

• Mixture-of-Experts
• handwriting trajectory reconstruction
• IMU sensors
• touching and hovering modeling
• benchmark dataset