Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards

2026-07-15

AI Summary

This work addresses the lack of generalization guarantees in reinforcement learning with verifiable rewards (RLVR) for large language model fine-tuning by introducing, for the first time, a PAC-Bayes compression bound into the RLVR framework. To handle the stochasticity of token generation, the approach leverages Gumbel-max reparameterization and proposes a Progressive RLVR training paradigm that integrates online policy distillation, TinyLoRA, and model quantization. The method achieves the first non-vacuous generalization bound at the billion-parameter scale, yielding a model compression ratio of 14,796× across mathematical reasoning, programming, commonsense reasoning, and Text-to-SQL tasks. It retains 84–97% of standard LoRA’s performance, significantly outperforms the base model in terms of the generalization bound (by 9–51%), and closely approaches full fine-tuning performance with only a 6–11% gap.

Abstract

While reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning capabilities of large language models (LLMs), the generalizability of the resulting models remains poorly understood. In this work, we establish the first non-vacuous generalization bounds for parameter-efficient RLVR fine-tuning at the billion-parameter scale. Our approach adapts PAC-Bayes compression bounds to this setting, and addresses the inherent stochasticity of token generation by applying the Gumbel-max reparameterization trick. To operationalize these bounds, we propose the Progressive RLVR framework, which integrates RLVR with on-policy distillation, TinyLoRA, and model quantization. Progressive RLVR empirically retains 84-97% performance of standard LoRA fine-tuning while producing models that are 14,796x more compressible. We show that this framework yields non-vacuous generalization bounds in four domains: mathematical problem-solving, programming, general-knowledge reasoning, and Text-to-SQL. Our bounds exceed the accuracy of the base model by 9-51% and lie within 6-11% of the accuracy of the fine-tuned models.

Problem

• generalization bounds
• reinforcement learning
• verifiable rewards
• large language models
• PAC-Bayes

Innovation

• non-vacuous generalization bounds
• PAC-Bayes
• Gumbel-max reparameterization
• Progressive RLVR
• parameter-efficient fine-tuning