Meta Engine: A Unified Semantic Query Engine on Heterogeneous LLM-Based Query Systems

2026-02-02

AI Summary

This work addresses the fragmentation of current multimodal semantic query systems, which suffer from inconsistent APIs and a trade-off between specialization and generalization. To overcome these limitations, the authors propose a “query system over query systems” architecture that unifies heterogeneous specialized LLM-based query engines under a single semantic query engine. This framework leverages natural language parsing, operator generation, dynamic query routing, adapter integration, and result aggregation to orchestrate and coordinate diverse backend systems. The approach maintains strong performance on unimodal tasks while substantially enhancing cross-modal querying capabilities. Empirical evaluations demonstrate consistent improvements across multiple datasets, with F1 scores increasing by 3–6 times over existing baselines and up to 24 times in specific scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

With the increasingly use of multi-modal data, semantic query has become more and more demanded in data management systems, which is an important way to access and analyze multi-modal data. As unstructured data, most information of multi-modal data (text, image, video, etc) hides in the semantics, which cannot be accessed by the traditional database queries like SQL. Given the power of Large Language Model (LLM) in understanding semantics and processing natural language, in recent years several LLM-based semantic query systems have been proposed, to support semantic querying over unstructured data. However, this rapid growth has produced a fragmented ecosystem. Applications face significant integration challenges due to (1) disparate APIs of different semantic query systems and (2) a fundamental trade-off between specialization and generality. Many semantic query systems are highly specialized, offering state-of-the-art performance within a single modality but struggling with multi-modal data. Conversely, some"all-in-one"systems handle multiple modalities but often exhibit suboptimal performance compared to their specialized counterparts in specific modalities. This paper introduces Meta Engine, a novel"query system on query systems", designed to resolve those aforementioned challenges. Meta Engine is a unified semantic query engine that integrates heterogeneous, specialized LLM-based query systems. Its architecture comprises five key components: (1) a Natural Language (NL) Query Parser, (2) an Operator Generator, (3) a Query Router, (4) a set of Adapters, and (5) a Result Aggregator. In the evaluation, Meta Engine consistently outperforms all baselines, yielding 3-6x higher F1 in most cases and up to 24x on specific datasets.

Problem

• semantic query
• heterogeneous LLM-based systems
• multi-modal data
• query integration
• unstructured data

Innovation

• unified semantic query
• heterogeneous LLM systems
• multi-modal data
• query routing
• semantic integration