A Method for Learning Value Systems in Generative AI

2026-07-18

AI Summary

This work addresses the challenge that current generative AI systems lack explicit modeling of the multidimensional structure of human values, hindering interpretable and consistent value alignment. To bridge this gap, the authors propose a joint learning framework leveraging pairwise prompt-response preference data, introducing value system learning into generative AI for the first time. Their approach constructs a value basis through a multi-objective reward model and represents the value system via weighted linear scalarization, augmented with a dynamic prioritization mechanism to ensure representational consistency. Evaluated across multiple preference datasets, the method outperforms existing baselines, achieving significantly enhanced interpretability and value awareness in alignment while incurring negligible performance degradation.

Abstract

Value-aware AI systems require explicit computational representations of human values (groundings) and their aggregation into value systems in order to align their decisions with ours. As such representations are difficult to elicit, value learning seeks to infer them by observing human behaviour. This work addresses the lack of grounded value learning methods in generative AI: existing approaches typically replicate human preferences without awareness of the multidimensional structure of value alignment, or lack principled value system elicitation methods. To address these gaps, we adapt a previously validated value system learning method to the generative AI setting, which, based on pairwise prompt-response preference data, simultaneously learns: i) an implementation of a grounding for a set of values given by a multi-objective reward model, and ii) a value system representation in the form of a weighted linear scalarization of the previous grounding model. To ensure that the learned value systems are based on coherent value representations, our algorithm dynamically prioritizes the grounding learning process. We evaluate the method against baselines and a contemporary method on prompt-response preference datasets. Results show competitive performance and minimal trade-offs against the baselines, while improving explainability.

Problem

• value learning
• generative AI
• value systems
• value alignment
• preference data

Innovation

• value learning
• generative AI
• value grounding
• multi-objective reward model
• value system