Boolean-aware Attention for Dense Retrieval

2025-03-03

AI Summary

Dense retrieval models struggle to capture the semantic nuances of Boolean queries (containing AND/OR/NOT operators). To address this, we propose the Boolean-aware Dense Retrieval (BDR) model. Its core is a Boolean-operator-driven dynamic attention mechanism: dedicated Boolean expert modules, combined with learnable gating strategies, explicitly modulate token-level attention weights according to each Boolean operator, and are seamlessly integrated into the BERT architecture. To our knowledge, BDR is the first dense retrieval method to enable fine-grained, interpretable semantic modeling of Boolean logic. Extensive experiments on multiple Boolean retrieval benchmarks demonstrate that BDR significantly outperforms standard BERT baselines, achieving average improvements of 12.7% in Recall@10 and MRR. These results validate that explicit Boolean semantic modeling provides a critical performance gain for dense retrieval.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Computer Vision: Image and Video Retrieval
• Knowledge Representation and Reasoning: Reasoning with Beliefs

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present Boolean-aware attention, a novel attention mechanism that dynamically adjusts token focus based on Boolean operators (e.g., and, or, not). Our model employs specialized Boolean experts, each tailored to amplify or suppress attention for operator-specific contexts. A predefined gating mechanism activates the corresponding experts based on the detected Boolean type. Experiments on Boolean retrieval datasets demonstrate that integrating BoolAttn with BERT greatly enhances the model's capability to process Boolean queries.

Problem

• Dynamic token focus adjustment using Boolean operators
• Specialized Boolean experts for operator-specific contexts
• Enhanced Boolean query processing with BERT integration

Innovation

• Dynamic token focus adjustment via Boolean operators
• Specialized Boolean experts for context-specific attention
• Predefined gating mechanism for expert activation