When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering

2026-08-06

AI Summary

Existing methods struggle to evaluate whether the consistency of vision-language models under perturbed inputs reflects genuine prediction correctness, primarily due to the semantic ambiguity of natural image perturbations and the absence of precise ground-truth answers. This work proposes RENDEQ, a framework that generates semantically equivalent image variants with exact, controllable answers by programmatically redrawing scientific figures. For the first time, it enables quantitative assessment of the relationship between consistency and accuracy under controlled conditions. Integrating multi-style factor control, cross-rendering consistency evaluation, and token-level confidence analysis, the study finds that re-rendering outperforms re-sampling, with consistency metrics consistently surpassing baselines. Errors are largely attributed to discrepancies across plotting libraries, and surprisingly, fine-tuning models on their own consensus systematically degrades accuracy—an unexpected and counterintuitive phenomenon.

Abstract

A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness. That coupling is rarely measured directly: natural-image perturbations preserve meaning only by assumption, and no exact answer key localizes errors. Scientific figures remove both obstacles, a figure is drawn from data by a program, so redrawing it yields images that are semantically equivalent by construction and share a programmatically exact answer. We build RENDEQ, a generator of such render-equivalence sets, and measure the coupling on three open-weight VLMs, checking every finding across three independent instantiations. Re-rendering beats resampling on both accuracy and reliability. Agreement beats an evidence-carrying baseline, mean token log-probability, on two of three models and ties on the third, reversing an intermediate, buggy replication traced to a rendering-pipeline failure. The dispersion behind this is concentrated in one style factor, the plotting library, more than double the next-largest factor and an order of magnitude above the noise floor. Fine-tuning on the model's own cross-render consensus inverts: accuracy falls in every one of five replication runs, the opposite sign to published results on natural images. Agreement certifies correctness only above a threshold set by how diffuse a model's errors are, and an objective that rewards agreement destroys exactly that diffuseness.

Problem

• agreement-accuracy coupling
• semantic-preserving perturbation
• visual language models
• render-equivalence
• model reliability

Innovation

• re-rendering
• agreement-accuracy coupling
• semantic equivalence
• visual language models
• reliability evaluation