Characterising Topic Familiarity and Query Specificity Using Eye-Tracking Data

2025-05-06

AI Summary

This study addresses the problem of inferring users’ topic familiarity and query specificity solely from eye-tracking data—without relying on query text, page content, or other contextual signals. Methodologically, it introduces pupil dilation magnitude and fixation velocity as novel, lightweight behavioral features reflecting higher-order cognitive states; proposes a human annotation protocol for query specificity tailored to question-answering scenarios; and employs gradient boosting and k-nearest neighbors classifiers for modeling. Contributions include: (1) demonstrating the discriminability of pure oculomotor signals for high-level cognitive dimensions, and (2) establishing the first fine-grained annotation framework specifically for query specificity. In a controlled laboratory study with 18 participants, the approach achieves a Macro F1 score of 71.25% for topic familiarity prediction and 60.54% for query specificity classification—empirically validating the feasibility and effectiveness of context-free inference.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Natural Language Processing: Question Answering

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Topic discovery and tracking
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Eye-tracking data has been shown to correlate with a user's knowledge level and query formulation behaviour. While previous work has focused primarily on eye gaze fixations for attention analysis, often requiring additional contextual information, our study investigates the memory-related cognitive dimension by relying solely on pupil dilation and gaze velocity to infer users' topic familiarity and query specificity without needing any contextual information. Using eye-tracking data collected via a lab user study (N=18), we achieved a Macro F1 score of 71.25% for predicting topic familiarity with a Gradient Boosting classifier, and a Macro F1 score of 60.54% with a k-nearest neighbours (KNN) classifier for query specificity. Furthermore, we developed a novel annotation guideline -- specifically tailored for question answering -- to manually classify queries as Specific or Non-specific. This study demonstrates the feasibility of eye-tracking to better understand topic familiarity and query specificity in search.

Problem

• Predict topic familiarity using pupil dilation and gaze velocity
• Classify query specificity without needing contextual information
• Develop annotation guidelines for question answering queries

Innovation

• Uses pupil dilation and gaze velocity
• Employs Gradient Boosting and KNN classifiers
• Develops novel query annotation guidelines