Detecting Pen-In-Air States from Video: A Proof-of-Concept Toward Complementary Handwriting Analysis

2026-06-01

AI Summary

This study addresses the limitation of conventional digitizing tablets in capturing pen-lift (pen-up) movements during handwriting, which hinders comprehensive assessment of writing impairments such as dysgraphia. To overcome this, the authors propose a non-invasive approach that infers pen-tip contact status solely from top-view video recordings. The method integrates YOLO-based pen-tip tracking, extracts kinematic features, and employs a machine learning classifier for frame-level pen-up detection. A novel dataset with manual annotations is introduced, and the work demonstrates for the first time that top-view video can serve as a low-cost, unobtrusive complement to digitizing tablets for detecting in-air writing gestures. Under leave-one-video-out cross-validation, the system achieves an F₂ score of 0.805 for pen-up event detection, indicating high recall reliability suitable for screening applications.

Abstract

Dynamic aspects of handwriting are critical for assessing developmental disorders such as dysgraphia and are typically captured using digitizing tablets. However, tablet-based sensing restricts analysis of Pen-Up behavior to a short proximity range above the writing surface, potentially missing high-lift in-air movements. As a proof of concept, we investigate whether top-view video can provide a complementary source of information for inferring pen-contact states without relying on tablet proximity sensing. We propose an interpretable hybrid pipeline combining pen-tip tracking using a YOLO-based detector with kinematic feature extraction and machine learning classification. A pilot dataset of diverse handwriting videos was manually annotated at the frame level and evaluation used a Leave-One-Video-Out (LOVO) protocol. The method achieved reliable event-level detection of Pen-Up segments, with an F_2 score up to 0.805, consistent with the emphasis on recall in a screening-oriented setting. These results support the feasibility of video-based Pen-Up detection as a low-cost and non-intrusive complement to digitizing tablets, and provide a foundation for future large-scale studies.

Problem

• Pen-Up detection
• handwriting analysis
• dysgraphia
• in-air movements
• digitizing tablets

Innovation

• video-based pen tracking
• Pen-Up detection
• handwriting analysis
• YOLO-based detection
• kinematic features