Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Prediction

2025-09-06

AI Summary

Multimodal multitask prediction in clinical settings faces challenges from sample-level data heterogeneity—such as co-occurring structured rating scales and unstructured clinical text—and varying task interdependencies, compounded by pervasive missing values. Method: We propose the first sample-adaptive routing framework for unified multimodal multitask learning. Built upon a mixture-of-experts architecture, it jointly learns modality-specific processing paths (for raw/fused textual and numerical features) and task-sharing strategies (dynamically assigning shared or task-specific prediction heads), enabling personalized information flow. The model is trained end-to-end to yield interpretable awareness of modality importance and task relationships. Results: Evaluated on synthetic data and real-world psychotherapy transcripts, our method significantly outperforms fixed multitask and single-task baselines in predicting depression and anxiety symptom severity. It demonstrates improved personalization and cost-effectiveness for clinical interventions, validating its practical utility in mental health applications.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

We propose a unified framework for adaptive routing in multitask, multimodal prediction settings where data heterogeneity and task interactions vary across samples. Motivated by applications in psychotherapy where structured assessments and unstructured clinician notes coexist with partially missing data and correlated outcomes, we introduce a routing-based architecture that dynamically selects modality processing pathways and task-sharing strategies on a per-sample basis. Our model defines multiple modality paths, including raw and fused representations of text and numeric features and learns to route each input through the most informative expert combination. Task-specific predictions are produced by shared or independent heads depending on the routing decision, and the entire system is trained end-to-end. We evaluate the model on both synthetic data and real-world psychotherapy notes predicting depression and anxiety outcomes. Our experiments show that our method consistently outperforms fixed multitask or single-task baselines, and that the learned routing policy provides interpretable insights into modality relevance and task structure. This addresses critical challenges in personalized healthcare by enabling per-subject adaptive information processing that accounts for data heterogeneity and task correlations. Applied to psychotherapy, this framework could improve mental health outcomes, enhance treatment assignment precision, and increase clinical cost-effectiveness through personalized intervention strategies.

Problem

• Adaptive routing for multimodal multitask prediction with heterogeneous data
• Dynamic modality selection and task-sharing strategies per sample
• Addressing missing data and task correlations in personalized healthcare

Innovation

• Dynamic routing selects modality pathways per sample
• Learns to route inputs through optimal expert combinations
• Task predictions use shared or independent routing heads