InkFM: A Foundational Model for Full-Page Online Handwritten Note Understanding

2025-03-29

AI Summary

The absence of a unified foundation model hinders holistic understanding of full-page online handwritten notes. Method: We propose the first vision-language foundation model for multilingual handwritten documents, integrating multilingual OCR, mathematical formula recognition, and page-structure parsing (text/drawing segmentation) within a single homogeneous architecture. Our approach unifies document image encoding, serialized layout modeling, and script-adaptive decoding heads, supporting zero-shot text-line segmentation and LoRA-based fine-tuning for cross-task generalization. Results: Our model achieves superior zero-shot text-line segmentation performance over baselines such as docTR. With lightweight fine-tuning, it attains state-of-the-art results across five major handwritten datasets—DeepWriting, CASIA, SCUT, Mathwriting, and QuickDraw—demonstrating comprehensive support for 28 scripts, mathematical formula recognition, and page-element parsing.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Tablets and styluses are increasingly popular for taking notes. To optimize this experience and ensure a smooth and efficient workflow, it's important to develop methods for accurately interpreting and understanding the content of handwritten digital notes. We introduce a foundational model called InkFM for analyzing full pages of handwritten content. Trained on a diverse mixture of tasks, this model offers a unique combination of capabilities: recognizing text in 28 different scripts, mathematical expressions recognition, and segmenting pages into distinct elements like text and drawings. Our results demonstrate that these tasks can be effectively unified within a single model, achieving SoTA text line segmentation out-of-the-box quality surpassing public baselines like docTR. Fine- or LoRA-tuning our base model on public datasets further improves the quality of page segmentation, achieves state-of the art text recognition (DeepWriting, CASIA, SCUT, and Mathwriting datasets) and sketch classification (QuickDraw). This adaptability of InkFM provides a powerful starting point for developing applications with handwritten input.

Problem

• Develops InkFM for full-page handwritten note analysis
• Unifies text, math, and drawing recognition in one model
• Achieves state-of-the-art performance across multiple datasets

Innovation

• Unified model for multi-task handwritten analysis
• Supports 28 scripts and math expressions recognition
• Achieves SoTA in segmentation and text recognition