Predicting Alignment Generalization with Value Representations

2026-10-08

AI Summary

This study addresses the challenge of "alignment generalization," wherein the behavior of fine-tuned large language models (LLMs) in unseen scenarios remains difficult to predict. We propose an activation-representation-based task for predicting alignment generalization and systematically analyze how 66 distinct values influence fine-tuning outcomes. Our findings reveal that internal activation representations predict fine-tuning effects more accurately than textual descriptions. Building on this insight, we construct the first value taxonomy and shared value space for LLMs. Large-scale empirical evaluations demonstrate a significant correlation between activation-based value similarity and model robustness (r=0.45). These results provide a quantifiable scientific foundation for principled LLM behavioral design.

Abstract

LLM developers post-train their models to exhibit prosocial values and behavioral traits, which are enumerated in an alignment target. However, while recent post-training developments have yielded models that score highly on alignment evaluations, training models on sets of narrow behaviors still influences their behavior across unseen contexts and environments in unexpected ways. In this paper, we establish the task of alignment generalization prediction, i.e., predicting how fine-tuning a model to follow a given value changes its behavior across a wide range of held-out values. We conduct a large-scale analysis of alignment generalization effects across 66 values found in modern alignment targets, and benchmark representational techniques on the alignment generalization prediction task. We find that representations based on model activations when applying values in context significantly outperform methods based on textual descriptions of the values. Specifically, the best activations-based methods achieve correlations of 0.45 with our generalization matrix, compared with 0.05 from description-based baselines. We then show the applicability of representations that predict alignment generalization toward downstream tasks by using them to measure how similar the values in a multi-value alignment target are, which we find is significantly correlated with model robustness. Finally, we show initial evidence towards a shared, model-independent value space, which we use to develop the first taxonomy of LLM values grounded in empirical generalization dynamics. Our work demonstrates the importance of studying value generalization in LLMs and its application toward the more empirical design and training of model behavior.

Problem

• Alignment Generalization
• Value Representations
• Large Language Models
• Post-training
• Behavior Prediction

Innovation

• Alignment Generalization
• Value Representations
• Activation-based Methods
• Shared Value Space
• LLM Taxonomy