Multimodal Evaluation of Russian-language Architectures

2025-11-19

AI Summary

The absence of Russian-language multimodal benchmarks hinders systematic evaluation of multimodal large language models (MLLMs) regarding their capabilities, limitations, and risks. Method: We introduce Mera Multi—the first open-source, Russian-centric multimodal evaluation framework—covering text, image, audio, and video modalities. It comprises 18 original tasks designed for both general-purpose and modality-specific architectures. We construct culturally adapted Russian multimodal datasets and propose a unified taxonomy for multimodal competencies. Adopting an instruction-based benchmarking paradigm, we standardize prompting templates and evaluation metrics, while integrating watermarking, licensing, and leakage-prevention mechanisms to ensure data security and cross-model (open- and closed-source) comparability. Contribution/Results: Mera Multi establishes a standardized, reproducible evaluation baseline for Russian multimodal research and provides a transferable benchmarking methodology applicable to other Slavic languages.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, and risks remain insufficiently understood. To address these issues, particularly in the context of the Russian language, where no multimodal benchmarks currently exist, we introduce Mera Multi, an open multimodal evaluation framework for Russian-spoken architectures. The benchmark is instruction-based and encompasses default text, image, audio, and video modalities, comprising 18 newly constructed evaluation tasks for both general-purpose models and modality-specific architectures (image-to-text, video-to-text, and audio-to-text). Our contributions include: (i) a universal taxonomy of multimodal abilities; (ii) 18 datasets created entirely from scratch with attention to Russian cultural and linguistic specificity, unified prompts, and metrics; (iii) baseline results for both closed-source and open-source models; (iv) a methodology for preventing benchmark leakage, including watermarking and licenses for private sets. While our current focus is on Russian, the proposed benchmark provides a replicable methodology for constructing multimodal benchmarks in typologically diverse languages, particularly within the Slavic language family.

Problem

• Evaluating multimodal Russian-language AI models lacking existing benchmarks
• Assessing intelligence and limitations of multimodal language architectures systematically
• Creating culturally-aware evaluation framework for Slavic language AI capabilities

Innovation

• Open multimodal evaluation framework for Russian
• Instruction-based benchmark with four modalities
• Methodology preventing leakage via watermarking and licensing