Unsupervised Document and Template Clustering using Multimodal Embeddings

2025-06-13

AI Summary

This work addresses the challenge in unsupervised document clustering of simultaneously distinguishing document categories (e.g., invoices vs. purchase orders) and fine-grained templates within each category. We propose a multi-modal embedding-based fine-grained clustering framework that fuses representations from pre-trained models: textual (SBERT), layout-language (LayoutLMv1/v3), visual (DiT, Donut), and document-image alignment (ColPali). The resulting joint embeddings drive dual-level clustering—document-category and template-level—using k-Means and DBSCAN. To our knowledge, this is the first systematic empirical validation of multi-modal embeddings for unsupervised document organization, revealing the representational capabilities and limitations of diverse models for structured document understanding. Experiments demonstrate significant improvements in clustering accuracy and template discrimination over unimodal baselines. The framework provides a reusable pipeline and an empirical benchmark for intelligent document processing.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

This paper investigates a novel approach to unsupervised document clustering by leveraging multimodal embeddings as input to traditional clustering algorithms such as $k$-Means and DBSCAN. Our method aims to achieve a finer-grained document understanding by not only grouping documents at the type level (e.g., invoices, purchase orders), but also distinguishing between different templates within the same document category. This is achieved by using embeddings that capture textual content, layout information, and visual features of documents. We evaluated the effectiveness of this approach using embeddings generated by several state-of-the-art pretrained multimodal models, including SBERT, LayoutLMv1, LayoutLMv3, DiT, Donut, and ColPali. Our findings demonstrate the potential of multimodal embeddings to significantly enhance document clustering, offering benefits for various applications in intelligent document processing, document layout analysis, and unsupervised document classification. This work provides valuable insight into the advantages and limitations of different multimodal models for this task and opens new avenues for future research to understand and organize document collections.

Problem

• Unsupervised clustering of documents using multimodal embeddings
• Distinguishing templates within the same document category
• Evaluating multimodal models for document understanding enhancement

Innovation

• Multimodal embeddings for document clustering
• Combining textual, layout, and visual features
• Evaluating state-of-the-art pretrained multimodal models