Mesh-RFT: Enhancing Mesh Generation via Fine-grained Reinforcement Fine-Tuning

2025-05-22

AI Summary

Pretrained 3D mesh generation models suffer from local structural distortions due to dataset bias and struggle to capture face-level details under global reinforcement learning. To address this, we propose a fine-grained reinforcement fine-tuning framework that pioneers face-level RL granularity. Our approach introduces Masked Direct Preference Optimization (M-DPO) and a topology-aware dual-metric scoring system—Balanced Edge Ratio (BER) and Topology Score (TS)—enabling quality-driven local correction while preserving global consistency. Experiments demonstrate that our method reduces Hausdorff Distance (HD) by 24.6% and improves TS by 3.8% over baseline models. Compared to global DPO, it further lowers HD by 17.4% and boosts TS by 4.9%, achieving state-of-the-art performance for production-grade 3D mesh generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Existing pretrained models for 3D mesh generation often suffer from data biases and produce low-quality results, while global reinforcement learning (RL) methods rely on object-level rewards that struggle to capture local structure details. To address these challenges, we present extbf{Mesh-RFT}, a novel fine-grained reinforcement fine-tuning framework that employs Masked Direct Preference Optimization (M-DPO) to enable localized refinement via quality-aware face masking. To facilitate efficient quality evaluation, we introduce an objective topology-aware scoring system to evaluate geometric integrity and topological regularity at both object and face levels through two metrics: Boundary Edge Ratio (BER) and Topology Score (TS). By integrating these metrics into a fine-grained RL strategy, Mesh-RFT becomes the first method to optimize mesh quality at the granularity of individual faces, resolving localized errors while preserving global coherence. Experiment results show that our M-DPO approach reduces Hausdorff Distance (HD) by 24.6% and improves Topology Score (TS) by 3.8% over pre-trained models, while outperforming global DPO methods with a 17.4% HD reduction and 4.9% TS gain. These results demonstrate Mesh-RFT's ability to improve geometric integrity and topological regularity, achieving new state-of-the-art performance in production-ready mesh generation. Project Page: href{https://hitcslj.github.io/mesh-rft/}{this https URL}.

Problem

• Improves 3D mesh generation quality by addressing data biases
• Enhances local structure details via fine-grained reinforcement learning
• Optimizes mesh quality at individual face level for better accuracy

Innovation

• Fine-grained RL with M-DPO for localized refinement
• Topology-aware scoring system with BER and TS
• Optimizes mesh quality at individual face level