🤖 AI Summary
Multimodal entity linking faces significant challenges in simultaneously achieving high accuracy, computational efficiency, and storage efficiency. This work proposes FAST-MEL, a lightweight encoder architecture that constructs unified, compact, fixed-length multimodal vector representations for both knowledge base entities and text–vision mentions, enabling efficient indexing and matching. By doing so, FAST-MEL maintains state-of-the-art entity linking accuracy while accelerating inference by three orders of magnitude and reducing storage overhead by an order of magnitude compared to the fastest existing system. This represents the first approach to successfully overcome the longstanding trade-off among these three critical objectives, demonstrating that they can be jointly optimized without compromising performance.
📝 Abstract
Multimodal entity linking (MEL) is the task that consists of matching textual and visual mentions of entities in unstructured data to their corresponding entities in a knowledge base (KB). To be effective in large-scale practical settings, MEL systems must meet three objectives: high linking accuracy, computational efficiency, and storage efficiency, i.e., a compact yet efficient index of the KB. In this paper, we highlight that state-of-the-art systems fail to simultaneously satisfy these 3 requirements. To meet this three-fold objective, we propose FAST-MEL, a lightweight encoder-based MEL solution that relies on a novel and compact fixed-size vectorized representation of both the textual and visual information of each entity or mention. It matches the accuracy of the best systems but performs three orders of magnitude faster. It also consumes one order of magnitude less storage than the fastest systems.