Chitrarth: Bridging Vision and Language for a Billion People

2025-02-21

AI Summary

Existing multimodal foundation models heavily rely on English or high-resource European language data, resulting in poor generalization to low- and medium-resource languages—particularly in India’s linguistically diverse settings. To address this, we introduce Chitrarth, the first inclusive vision-language model tailored to ten major Indian languages. Our approach features: (1) a unified multilingual vision-language architecture integrating a state-of-the-art multilingual large language model with a visual encoder; (2) BharatBench—the first open, India-specific multimodal evaluation benchmark; and (3) training on high-quality, multilingual image–text alignment data. Experiments demonstrate that Chitrarth achieves state-of-the-art performance across multiple low-resource language benchmarks while maintaining competitive accuracy on English tasks. Notably, it significantly advances fairness and effectiveness in multilingual visual understanding, bridging critical gaps in cross-lingual multimodal representation learning.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent multimodal foundation models are primarily trained on English or high resource European language data, which hinders their applicability to other medium and low-resource languages. To address this limitation, we introduce Chitrarth (Chitra: Image; Artha: Meaning), an inclusive Vision-Language Model (VLM), specifically targeting the rich linguistic diversity and visual reasoning across 10 prominent Indian languages. Our model effectively integrates a state-of-the-art (SOTA) multilingual Large Language Model (LLM) with a vision module, primarily trained on multilingual image-text data. Furthermore, we also introduce BharatBench, a comprehensive framework for evaluating VLMs across various Indian languages, ultimately contributing to more diverse and effective AI systems. Our model achieves SOTA results for benchmarks across low resource languages while retaining its efficiency in English. Through our research, we aim to set new benchmarks in multilingual-multimodal capabilities, offering substantial improvements over existing models and establishing a foundation to facilitate future advancements in this arena.

Problem

• Multilingual Vision-Language Model
• Low-resource Indian languages
• Multimodal AI diversity improvement

Innovation

• Multilingual Vision-Language Model
• Integrates LLM with vision
• Introduces BharatBench evaluation framework