Do LLMs and Humans Find the Same Questions Difficult? A Case Study on Japanese Quiz Answering

2025-11-15

AI Summary

This study investigates the alignment between large language models (LLMs) and human annotators in judging question difficulty within Japanese quiz-bowl–style问答. Using a novel, manually curated Japanese question-answering dataset, we employ multi-prompting strategies to elicit responses from LLMs and systematically compare their accuracy against human performance along two key dimensions: whether the answer is covered in Wikipedia and whether the answer is numeric. Results reveal that LLMs significantly underperform humans on questions involving Wikipedia-uncovered knowledge and those requiring numeric reasoning—highlighting their heavy reliance on training data coverage and fundamental limitations in flexible numerical computation. To our knowledge, this is the first empirical study to uncover structural misalignment in difficulty perception between LLMs and humans in a Japanese quiz context. The work provides a reproducible analytical framework for probing model knowledge boundaries and informing robust prompt engineering.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

LLMs have achieved performance that surpasses humans in many NLP tasks. However, it remains unclear whether problems that are difficult for humans are also difficult for LLMs. This study investigates how the difficulty of quizzes in a buzzer setting differs between LLMs and humans. Specifically, we first collect Japanese quiz data including questions, answers, and correct response rate of humans, then prompted LLMs to answer the quizzes under several settings, and compare their correct answer rate to that of humans from two analytical perspectives. The experimental results showed that, compared to humans, LLMs struggle more with quizzes whose correct answers are not covered by Wikipedia entries, and also have difficulty with questions that require numerical answers.

Problem

• Comparing difficulty patterns between LLMs and humans in Japanese quiz answering
• Investigating if human-difficult questions are also challenging for language models
• Analyzing LLM performance on Wikipedia-independent and numerical answer questions

Innovation

• Analyzing LLM performance on Japanese quiz data
• Comparing human and LLM difficulty patterns systematically
• Identifying Wikipedia coverage impact on LLM accuracy