Problem
Research questions and friction points this paper is trying to address.
Program-to-Visual gap
Image and video generation
Executable code
Multimodal large language models
Visual quality evaluation
MaLiang-Harness: A Programmable Path to Image and Video Generation
🤖 AI Summary
This study addresses the program-to-visual inconsistency arising when multimodal large language models (MLLMs) generate code and corresponding visual outputs. To mitigate this issue, it proposes a unified framework incorporating three core mechanisms—Persistent State (PEG), Traceable Generation Process (TGP), and Revision Awareness (REV)—which collectively establish a shared revision reference system to coordinate planning, execution, and feedback. By integrating MLLM-driven executable program generation, rendering backends, and closed-loop verification techniques, the framework enables controllable image and video synthesis. Extensive benchmark evaluations demonstrate that GPT-6-Astra achieves a 100% generation success rate within this framework. Furthermore, the experimental results reveal a significant discrepancy between general capability scores and actual visual generation performance, highlighting critical limitations in current evaluation paradigms for assessing the visual fidelity of MLLMs.
📝 Abstract
Executable programs offer explicit control over how images and videos are constructed, but generating runnable code is only the beginning of visual creation. A program can execute correctly while violating the requested composition, appearance, or motion. We define this discrepancy as the Program-to-Visual (P2V) gap and introduce MaLiang-Harness, a unified framework for organizing MLLM-driven visual generation into a persistent process of construction, inspection, and revision. Its central design is to make the evolving visual program, its construction history, and its verification share a common revision reference. We define the Persistent Executable Generation (PEG) state as preserving programs and task context. Traceable Generation Process (TGP) connects edits to rendered evidence, and Revision-aware Editing and Verification (REV) supports restoration and checks the current revision before completion. Together, these mechanisms coordinate planning, execution, and visual feedback across rendering backends. We evaluate 11 powerful closed-source MLLMs on MaLiang-IBench and four on MaLiang-VBench, measuring generation success, visual quality, and computational cost. GPT-6-Astra achieves 100% generation success on both benchmarks, with 96.0% of image tasks and 76.9% of video tasks meeting all quality thresholds. The comparison also reveals a mismatch between general capability scores and visual generation performance, with similarly scored models differing substantially in their ability to satisfy visual requirements. MaLiang-Harness provides a systematic basis for studying how MLLMs translate executable code into visual outcomes, exposing both the potential of programmable generation and the limitations of general benchmarks as predictors of this ability. The project is available at https://github.com/gulucaptain/MaLiang-Harness.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Program-to-Visual gap
Persistent Executable Generation
Traceable Generation Process
Revision-aware Editing and Verification
Programmable visual generation
🔎 Similar Papers
No similar papers found.
