Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository

2025-02-28

AI Summary

Standardized reading comprehension assessments often report only item p-values (proportion correct), lacking Item Response Theory (IRT) difficulty parameters essential for psychometric analysis. Method: We constructed a high-quality, manually annotated item bank covering New York and Texas grade 3–8 reading tests from 2017–2023. Our approach innovatively integrates three human-annotated dimensions—linguistic features, test metadata, and contextual features—alongside embeddings from BERT, ModernBERT, and Llama for multimodal representation learning. Contribution/Results: Experiments show that linguistic features or LLM embeddings alone achieve comparable performance (RMSE = 0.52), significantly improving upon the baseline (0.92) and demonstrating feasibility for lightweight deployment. The correlation between predicted and ground-truth IRT difficulty reaches 0.77. The model is open-sourced, establishing a reproducible, scalable paradigm for IRT difficulty estimation in educational measurement.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP ModelsIntelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsSearch and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Prediction of item difficulty based on its text content is of substantial interest. In this paper, we focus on the related problem of recovering IRT-based difficulty when the data originally reported item p-value (percent correct responses). We model this item difficulty using a repository of reading passages and student data from US standardized tests from New York and Texas for grades 3-8 spanning the years 2017-23. This repository is annotated with meta-data on (1) linguistic features of the reading items, (2) test features of the passage, and (3) context features. A penalized regression prediction model with all these features can predict item difficulty with RMSE 0.52 compared to baseline RMSE of 0.92, and with a correlation of 0.77 between true and predicted difficulty. We supplement these features with embeddings from LLMs (ModernBERT, BERT, and LlAMA), which marginally improve item difficulty prediction. When models use only item linguistic features or LLM embeddings, prediction performance is similar, which suggests that only one of these feature categories may be required. This item difficulty prediction model can be used to filter and categorize reading items and will be made publicly available for use by other stakeholders.

Problem

• Predicting item difficulty for reading comprehension using text content.
• Recovering IRT-based difficulty from item p-value data.
• Improving prediction with linguistic, test, and context features.

Innovation

• Annotated repository with linguistic, test, context features.
• Penalized regression model predicts item difficulty accurately.
• LLM embeddings marginally enhance difficulty prediction performance.