SCP-NL2TL: Selective Conformal Prediction with Semantic Verification for Natural Language to Temporal Logic Specifications

2026-08-05

AI Summary

This work addresses the safety risks posed by unreliable translations from natural language to formal specifications in safety-critical systems, where erroneous outputs are often generated without indication of uncertainty. To mitigate this issue, the authors propose an uncertainty-aware selective translation framework that simultaneously generates temporal logic specifications—such as STL, LTL, or SpaTiaL—and assesses their reliability. The framework actively abstains from producing outputs for low-confidence or out-of-distribution inputs. It integrates two complementary reliability signals: back-translation semantic fidelity and semantic dispersion across multiple candidate translations. By combining selective conformal prediction with conformal anomaly detection, the approach achieves distribution-free risk control. Experimental results demonstrate that the proposed method significantly enhances translation reliability and robustness, substantially reduces the acceptance rate of incorrect specifications, and accurately identifies out-of-distribution instructions.

Abstract

Translating natural language instructions into machine-interpretable formal specifications enables robots and autonomous systems to plan, reason, and formally verify their behavior. However, existing translation models typically generate a specification for every input, even when the result is unreliable or fails to capture the user's intent, creating risks in safety-critical applications. Inspired by selective conformal prediction, we propose a selective translation framework that not only generates formal specifications but also determines when they can be trusted. Reliability is scored by two complementary black-box signals, the fidelity of the specification back-translated into natural language and the dispersion of repeated translations under exact semantic equivalence, which fail on different errors and jointly separate incorrect translations more sharply than either alone. Conformal risk control calibrates this score into a decision that accepts a specification or abstains, with a distribution-free bound on the rate at which incorrect specifications are accepted for execution, and a conformal anomaly detector on instruction embeddings screens out-of-distribution inputs before any translation is attempted. The proposed framework is general across formal specification languages, with experiments on Signal Temporal Logic (STL), Linear Temporal Logic (LTL), and geometric Spatio-Temporal Logic (SpaTiaL) demonstrating improved translation reliability, robustness under the evaluated cross-tier shifts, and effective uncertainty-aware abstention. This work establishes a foundation for trustworthy natural language interfaces by enabling AI systems to recognize when generated specifications may not be reliable.

Problem

• Natural Language to Formal Specification
• Translation Reliability
• Safety-Critical Systems
• Uncertainty Awareness
• Out-of-Distribution Detection

Innovation

• Selective Conformal Prediction
• Natural Language to Formal Specification
• Semantic Verification
• Conformal Risk Control
• Uncertainty-aware Abstention