Fenchel Tilting: Weighted Correction for Efficient Finetuning of Generative Models

2026-09-30

AI Summary

This study addresses the challenge of simultaneously achieving generalizability and computational efficiency in generative model fine-tuning by proposing the FTFC method. This approach decouples utility optimization from model fitting, leveraging Fenchel duality theory to support general f-divergence penalties. By freezing model weights to avoid backpropagating through sampling trajectories, it enables efficient weighted-correction fine-tuning. The proposed technical framework is compatible with diffusion models, flow matching, and importance-weighted denoising. Experimental results demonstrate that FTFC outperforms existing baselines on both image and molecular generation benchmarks while achieving up to a 20-fold improvement in computational efficiency.

Abstract

Adapting a pretrained generative model to an arbitrary preference expressed as a utility function underlies reward alignment, guided design, and constraint satisfaction, enabling diverse applications. Existing fine-tuning methods trade off generality against computational cost: they either restrict the family class of supported preferences to keep optimization simple or preserve generality at the expense of efficiency. We introduce Fenchel Tilt Flow Control (FTFC), which decouples utility optimization from generative-model fitting. FTFC first optimizes for a target distribution by jointly fitting an effective reward and density-ratio weights on pretrained samples. Method combines the utility's variational structure with Fenchel duality, supporting general $f$-divergence penalties that determine how rewards are transformed into an distribution-correction weights. These weights are then frozen and used to modify a diffusion or flow model in a single stage of importance-weighted denoising or flow matching, without differentiating through sampling trajectories. We establish exact duality for concave utilities under suitable conditions and show that weighted fitting reproduces the optimal target distribution for a given utility. Across image and molecule generation benchmarks, FTFC improves over baselines on diverse preference functions, while also being up to $20\times$ more efficient. roposed method enables adaptation beyond expected-reward maximization without complex optimization, while preserving robustness for more general class of the utility functions compared to baselines.

Problem

• generative model finetuning
• preference alignment
• utility optimization
• computational efficiency
• f-divergence

Innovation

• Fenchel duality
• importance-weighted fine-tuning
• f-divergence
• flow matching
• reward alignment