Beyond Multiple Choice: Evaluating Steering Vectors for Adaptive Free-Form Summarization

2025-05-30

AI Summary

This paper addresses the underexplored controllability of steering vectors in free-form text generation, presenting the first systematic evaluation of their adaptive control over multidimensional attributes—including topic focus, sentiment, toxicity, and readability—in abstractive summarization. Methodologically, it employs activation-layer bias-based steering vectors integrated with prompt engineering, conducting experiments on the NEWTS dataset and establishing a multidimensional quantitative evaluation framework. Key contributions are threefold: (1) It reveals that strong steering enhances attribute control fidelity but degrades textual coherence and practical utility; (2) it proposes a synergistic steering–prompting strategy that achieves optimal trade-offs between control precision and generation quality at moderate intervention strength; (3) it advances beyond conventional multiple-choice evaluation paradigms by establishing a novel, generation-oriented benchmark for assessing steering vector efficacy in open-ended generative settings.

Technology Category

• Natural Language Processing: Generation
• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Steering vectors are a lightweight method for controlling text properties by adding a learned bias to language model activations at inference time. So far, steering vectors have predominantly been evaluated in multiple-choice settings, while their effectiveness in free-form generation tasks remains understudied. Moving"Beyond Multiple Choice,"we thoroughly evaluate the effectiveness of steering vectors in adaptively controlling topical focus, sentiment, toxicity, and readability in abstractive summaries of the NEWTS dataset. We find that steering effectively controls the targeted summary properties, but high steering strengths consistently degrade both intrinsic and extrinsic text quality. Compared to steering, prompting offers weaker control, while preserving text quality. Combining steering and prompting yields the strongest control over text properties and offers the most favorable efficacy-quality trade-off at moderate steering strengths. Our results underscore the practical trade-off between control strength and text quality preservation when applying steering vectors to free-form generation tasks.

Problem

• Evaluates steering vectors for adaptive free-form summarization control
• Assesses effectiveness in controlling topic, sentiment, toxicity, readability
• Examines trade-off between control strength and text quality

Innovation

• Steering vectors control text properties adaptively
• Combining steering and prompting enhances control
• Moderate steering balances efficacy and quality