Q-Steer: Action-Value Guidance for Molecular Policy Optimization

2026-07-28

AI Summary

This work addresses the myopic optimization problem in molecular design, where sparse rewards are only available upon complete molecule generation, making it difficult for policy-based methods to evaluate the contribution of intermediate actions. To overcome this, the authors propose Q-Steer, a general framework that guides token selection during rollout by leveraging an offline-trained and frozen Prefix-Action Value Scorer (PAVS-Q). This scorer estimates the expected future reward for each candidate SMILES token given the current prefix, and the normalized value advantage is incorporated into the sampling logits to steer generation. Without modifying the online optimizer or increasing the oracle query budget, Q-Steer consistently improves performance across diverse policy and language model backbones. On the PMO23 benchmark with a fixed budget of 10,000 oracle calls, it achieves gains in mean valid unique score ranging from +0.033 to +0.049 across all eight backbone–optimizer combinations, outperforming baselines in 18–20 out of 20 tasks per setting.

Abstract

Oracle-limited molecular optimization gives reward only after a complete molecule is generated, while each rollout requires many local next-token decisions. This delayed-feedback interface makes molecular policy optimization myopic: an optimizer can learn that a molecule was good without knowing which intermediate actions made it good. We introduce Q-Steer, a rollout-time action-value steering primitive for molecular language models. Q-Steer uses an offline-trained and frozen prefix-action value scorer, PAVS-Q, that estimates the downstream reward of taking a candidate next token under a partial SMILES prefix, then adds a normalized value bonus to sampling logits. The optimizer update rule and online oracle budget are unchanged; the claim is fixed-online-oracle performance, not equal total compute. On PMO23 with a fixed 10,000-call online budget, complete factorial studies across two molecular language-model backbones and four optimizers show that Q-Steer improves mean valid-unique score in all eight backbone-optimizer cells, with positive macro mean-score gains between +0.033 and +0.049 and 18-20 task wins per cell. Mechanism controls show that action identity matters: prefix-broadcast values are nearly neutral, while shuffled action values harm performance. These results support Q-Steer as a reusable rollout-time action-value wrapper that improves average molecular optimization reward across optimizer families and policy backbones without changing the online oracle budget.

Problem

• molecular optimization
• delayed feedback
• action-value guidance
• policy optimization
• oracle-limited

Innovation

• Q-Steer
• action-value guidance
• molecular optimization
• offline value scorer
• SMILES generation