Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models

2024-10-10

International Conference on Learning Representations

AI Summary

Multimodal large language models (MLLMs) suffer from low reliability, and existing split conformal prediction (SCP) methods struggle to adapt to open-ended generation and dynamic environments. Method: We propose TRON, a two-stage risk-control framework that first generates a minimal response set and then filters high-quality responses via self-consistency, unifying support for both open- and closed-ended tasks under dual risk-level constraints. Contribution/Results: We innovatively extend SCP to open-ended generation, uncovering and leveraging semantic redundancy within prediction sets; we introduce average set size as a novel evaluation metric. Experiments across four VideoQA benchmarks and eight MLLMs demonstrate strict adherence to user-specified dual risk bounds. Moreover, deduplication-optimized response sets exhibit superior efficiency, stability, and cross-model robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Multimodal Large Language Models (MLLMs) exhibit promising advancements across various tasks, yet they still encounter significant trustworthiness issues. Prior studies apply Split Conformal Prediction (SCP) in language modeling to construct prediction sets with statistical guarantees. However, these methods typically rely on internal model logits or are restricted to multiple-choice settings, which hampers their generalizability and adaptability in dynamic, open-ended environments. In this paper, we introduce TRON, a two-step framework for risk control and assessment, applicable to any MLLM that supports sampling in both open-ended and closed-ended scenarios. TRON comprises two main components: (1) a novel conformal score to sample response sets of minimum size, and (2) a nonconformity score to identify high-quality responses based on self-consistency theory, controlling the error rates by two specific risk levels. Furthermore, we investigate semantic redundancy in prediction sets within open-ended contexts for the first time, leading to a promising evaluation metric for MLLMs based on average set size. Our comprehensive experiments across four Video Question-Answering (VideoQA) datasets utilizing eight MLLMs show that TRON achieves desired error rates bounded by two user-specified risk levels. Additionally, deduplicated prediction sets maintain adaptiveness while being more efficient and stable for risk assessment under different risk levels.

Problem

• Address trustworthiness issues in Multimodal Large Language Models
• Generalize risk control methods for open-ended and closed-ended scenarios
• Reduce semantic redundancy in prediction sets for better evaluation

Innovation

• Two-step framework TRON for risk control
• Novel conformal score for minimal response sets
• Nonconformity score ensures high-quality responses