ObfusQAte: A Proposed Framework to Evaluate LLM Robustness on Obfuscated Factual Question Answering

2025-08-10

AI Summary

This work addresses the lack of robustness evaluation for large language models (LLMs) on fact-based question answering under semantic obfuscation. We propose ObfusQAte, a novel evaluation framework, and ObfusQA, a dedicated benchmark dataset. ObfusQAte is the first to systematically construct a multi-level obfuscation suite covering three semantic perturbation types: named entity indirectization, distractor-induced confusion, and context overload. It employs a fine-grained, rule- and template-based perturbation generation methodology—including entity substitution, irrelevant information injection, and contextual redundancy. Extensive experiments demonstrate that mainstream LLMs suffer substantial performance degradation and significantly increased hallucination rates across all obfuscation categories. This work establishes the first systematic, reproducible benchmark and methodology for assessing LLM robustness under semantic obfuscation, revealing critical vulnerabilities in handling complex linguistic variations.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The rapid proliferation of Large Language Models (LLMs) has significantly contributed to the development of equitable AI systems capable of factual question-answering (QA). However, no known study tests the LLMs' robustness when presented with obfuscated versions of questions. To systematically evaluate these limitations, we propose a novel technique, ObfusQAte and, leveraging the same, introduce ObfusQA, a comprehensive, first of its kind, framework with multi-tiered obfuscation levels designed to examine LLM capabilities across three distinct dimensions: (i) Named-Entity Indirection, (ii) Distractor Indirection, and (iii) Contextual Overload. By capturing these fine-grained distinctions in language, ObfusQA provides a comprehensive benchmark for evaluating LLM robustness and adaptability. Our study observes that LLMs exhibit a tendency to fail or generate hallucinated responses when confronted with these increasingly nuanced variations. To foster research in this direction, we make ObfusQAte publicly available.

Problem

• Evaluates LLM robustness on obfuscated factual QA
• Tests LLM performance under multi-tiered obfuscation
• Measures LLM adaptability to nuanced language variations

Innovation

• Multi-tiered obfuscation framework for LLMs
• Tests robustness via Named-Entity Indirection
• Evaluates LLMs with Distractor Indirection