A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation

2026-07-30

AI Summary

Existing vision-language models treat the translation of hand-drawn UML sketches into PlantUML as a generic image-to-text task, disregarding the formal semantics of UML and consequently producing outputs requiring extensive manual correction. This work proposes a formalism-aware refinement approach that integrates supervised fine-tuning with Group Relative Policy Optimization and, for the first time, incorporates formal reward signals derived from XMI representations and control flow graphs to guide the generation of semantically correct PlantUML code. Experimental results demonstrate that the optimized model surpasses both open-source and certain commercial baselines in terms of compilability and translation quality, achieving performance on class diagram tasks comparable to that of more powerful commercial systems, thereby validating the efficacy of the proposed formal feedback mechanism.

Abstract

Handwritten UML sketches are common in early software design, but turning them into structured, analysable modelling artefacts still requires manual reconstruction. Vision-language models can generate PlantUML from diagram images, but prompt-based use treats this as image-to-text generation rather than structured model generation. We investigate formalism-aware rewards: feedback signals derived from analysable model representations rather than surface text. In a worked example, we adapt a vision-language model for handwritten UML-to-PlantUML generation using super-vised fine-tuning followed by Group Relative Policy Optimisation. Generated PlantUML is compared against target representations, using XMI for class diagrams and control-flow graphs for activity diagrams. Emerging results show that the adapted model improves compilability and conversion quality over the untuned open model and one proprietary baseline, while remaining competitive with a stronger proprietary baseline on class diagrams. The added benefit of the reward-guided stage remains open on the current held-out set. Error analysis and metric-validity results show that modelling acceptability is only partially captured, motivating rewards and evaluations that combine model analysis with human judgement.

Problem

• Handwritten UML
• PlantUML
• structured model generation
• formalism-aware reward
• model reconstruction

Innovation

• formalism-aware reward
• PlantUML generation
• vision-language model
• structured model generation
• Group Relative Policy Optimisation