Penquiry: A Pen-based Interactive In-situ Q&A System Leveraging LLMs

2026-09-17

AI Summary

为解决笔式设备与大型语言模型间交互鸿沟问题，Penquiry系统通过内容捕捉和问题自动补全方法，实现直接在数字材料上提问。

Abstract

Pen-based digital devices remain a preferred medium for active, cognitively engaging study. Concurrently, Large Language Models (LLMs) have become indispensable for self-directed learning, enabling students to clarify concepts. However, a fundamental interaction gap exists between the fluid, spatial nature of pen-based workflows and the discrete, keyboard-heavy requirements of LLMs. We present Penquiry, an in-situ question-and-answer system that bridges this gap by enabling learners to pose questions directly on digital study materials via a pen. We characterize two primary interaction challenges in this multimodal transition: a Referential Barrier, which hinders grounding fine-grained visual elements into the query context, and an Expressive Barrier, which forces learners to translate diverse, non-textual intents--such as equations and diagrams--into rigid, typed sentences. To resolve these, Penquiry introduces a mediation layer featuring Content Snapping for unambiguous referencing and Question Autocompletion to expand sparse ink keywords into rich semantic queries. Through two iterative user studies (N = 16 per study), we demonstrate that Penquiry significantly reduces the cognitive and physical overhead of inquiry compared to traditional interfaces, providing a new blueprint for pen-based, in-situ AI interaction

Problem

• Pen-based Interaction
• Referential Barrier
• Expressive Barrier
• In-situ Q&A
• Large Language Models

Innovation

• Content Snapping
• Question Autocompletion
• pen-based interaction