Problem
Research questions and friction points this paper is trying to address.
time-series pattern querying
natural language
sketch-based interaction
fuzzy pattern description
user intent expression
ShapeTalk: Combining Natural Language and Sketch for Time-Series Pattern Querying
🤖 AI Summary
This work addresses the limitations of existing time series pattern querying tools, which struggle to capture vague or composite user intents due to reliance on exact sketches or rigid filters. The authors propose a novel co-querying system that synergistically integrates natural language and sketching as complementary input modalities: natural language conveys semantic intent and compositional logic, while an editable sketch enables geometric refinement. These modalities are tightly coupled within a shared visual context to support iterative exploration. The system combines large language model–based semantic parsing, editable shape-based feature constraints, a multimodal interactive interface, and an efficient pattern-matching algorithm. User studies demonstrate that this approach significantly enhances the flexibility and usability of querying imprecise patterns—natural language lowers the entry barrier, and sketching provides an effective correction mechanism when semantic descriptions fall short.
📝 Abstract
Searching for time-series segments that match user-defined patterns is important in domains such as finance, climate science, and healthcare. However, existing visual query tools often struggle to support vague, composite, or fuzzy pattern descriptions, often requiring users to express their intent through precise sketches or rigid structured filters. We present ShapeTalk, a coordinated natural-language and sketch-based querying system for univariate time-series pattern search. Rather than treating text and sketch as a fused input stream, ShapeTalk uses them as complementary representations of analytic intent: natural language supports semantic and compositional pattern descriptions, while sketching supports direct geometric refinement. The two modalities are linked through a shared visual context, editable feature representations, and synchronized result views, enabling users to move between text and sketch during iterative query formulation. At its core is an LLM-based semantic parsing pipeline that translates free-form natural-language queries into interpretable and editable shape-feature constraints. We evaluate ShapeTalk through two usage scenarios, a user study with failure-case analysis, and an assessment of the LLM-based semantic parsing pipeline. The results show that ShapeTalk supports effective time-series pattern search, with natural language serving as an accessible entry point and sketching providing a complementary mechanism for refinement and recovery when textual specifications are insufficient.
Innovation
Methods, ideas, or system contributions that make the work stand out.
multimodal querying
natural language interface
sketch-based interaction
time-series pattern search
LLM-based semantic parsing
🔎 Similar Papers
No similar papers found.
