Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions

2025-08-16

AI Summary

This work exposes critical prompt injection robustness vulnerabilities in large language models (LLMs) when deployed as automated evaluators (“LLM-as-a-judge”), particularly against hidden malicious instructions embedded in real-world PDF documents. Method: We design a minimal yet effective evaluation paradigm: under zero-shot settings, mainstream LLMs are tasked with solving elementary arithmetic questions (e.g., “3 + 2 = ?”) presented as multiple-choice or true/false items—despite adversarial PDF formatting that subtly injects misleading prompts. Contribution/Results: Experiments reveal that even on syntactically unambiguous, low-complexity tasks, all tested models exhibit significant susceptibility to implicit prompt interference, with error rates sharply increasing—some miscomputing basic operations like 3 + 2. This is the first study to instantiate prompt injection attacks within authentic PDF documents, empirically demonstrating the severe fragility of LLM-based evaluators under input noise. The findings provide both a critical caution and a reproducible benchmark for building trustworthy AI evaluation frameworks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have recently demonstrated strong emergent abilities in complex reasoning and zero-shot generalization, showing unprecedented potential for LLM-as-a-judge applications in education, peer review, and data quality evaluation. However, their robustness under prompt injection attacks, where malicious instructions are embedded into the content to manipulate outputs, remains a significant concern. In this work, we explore a frustratingly simple yet effective attack setting to test whether LLMs can be easily misled. Specifically, we evaluate LLMs on basic arithmetic questions (e.g., "What is 3 + 2?") presented as either multiple-choice or true-false judgment problems within PDF files, where hidden prompts are injected into the file. Our results reveal that LLMs are indeed vulnerable to such hidden prompt injection attacks, even in these trivial scenarios, highlighting serious robustness risks for LLM-as-a-judge applications.

Problem

• Assessing LLM vulnerability to hidden prompt injection attacks
• Testing robustness on simple arithmetic multiple-choice questions
• Evaluating security risks in LLM-as-a-judge applications

Innovation

• Hidden prompt injection in PDFs
• Testing LLMs on simple arithmetic
• Revealing vulnerability in trivial scenarios