MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization

2026-01-12

arXiv.org

AI Summary

This work addresses the challenge of aligning open-domain large language models with multiple, often conflicting objectives—such as creativity and factuality—where static reward scalarization fails to adapt to varying task demands. The authors propose a novel metacognitive coordination mechanism that formulates reward scalarization as a dynamic latent policy. This is achieved through a lightweight Conductor network that infers task context via a semantic bottleneck derived from terminal hidden states, and dynamically adjusts objective weights using a contextual bandit. Within a bilevel optimization framework, group-wise relative advantage serves as the meta-reward signal. The approach consistently outperforms single-reward and static multi-objective baselines across seven benchmarks, effectively reducing redundant generation while preserving the efficiency of GRPO and enabling task-aware adaptive alignment.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Search and Optimization: Metareasoning and MetaheuristicsNatural Language Processing: (Large) Language Models

Application Category

Economics, Online Markets and Human Computation: Cost models of using LLMs in production systemsUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Group-Relative Policy Optimization (GRPO) has emerged as an efficient paradigm for aligning Large Language Models (LLMs), yet its efficacy is primarily confined to domains with verifiable ground truths. Extending GRPO to open-domain settings remains a critical challenge, as unconstrained generation entails multi-faceted and often conflicting objectives - such as creativity versus factuality - where rigid, static reward scalarization is inherently suboptimal. To address this, we propose MAESTRO (Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization), which introduces a meta-cognitive orchestration layer that treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck to perceive task-specific priorities. We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal. Across seven benchmarks, MAESTRO consistently outperforms single-reward and static multi-objective baselines, while preserving the efficiency advantages of GRPO, and in some settings even reducing redundant generation.

Problem

• reward optimization
• multi-objective alignment
• open-domain generation
• reward scalarization
• Large Language Models

Innovation

• meta-learning
• reward scalarization
• multi-objective optimization
• contextual bandit
• group-relative policy optimization