Less Is More: Measuring How LLM Involvement affects Chatbot Accuracy in Static Analysis

2026-04-23

AI Summary

This study addresses the lack of systematic evaluation regarding the involvement of large language models (LLMs) in natural language interfaces for static program analysis. It introduces LLM intervention strategy as an independent variable and comparatively evaluates three architectural approaches—direct query generation, structured intermediate representation (IR) generation, and tool-augmented agent frameworks—for translating natural language into CPGQL queries. Experiments leverage the Joern platform, four open-source LLMs spanning two model families and two scales, and a JSON-schema-constrained IR. Results demonstrate that the structured IR approach achieves 15–25 percentage points higher match accuracy than direct generation on larger models and outperforms the agent-based method with only one-eighth of the token cost. Smaller models, however, struggle with schema compliance. This work establishes a new paradigm for efficient and accurate LLM integration in formal domains.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models are increasingly used to make static analysis tools accessible through natural language, yet existing systems differ in how much they delegate to the LLM without treating the degree of delegation as an independent variable. We compare three architectures along a spectrum of LLM involvement for translating natural language to Joern's query language \cpgql{}: direct query generation (\approach{1}), generation of a schema-constrained JSON intermediate representation (\approach{2}), and tool-augmented agentic generation (\approach{3}). These are evaluated on a benchmark of 20 code analysis tasks across three complexity tiers, using four open-weight models in a 2\(\times\)2 design (two model families \(\times\) two scales), each with three repetitions. The structured intermediate representation (\approach{2}) achieves the highest result match rates, outperforming direct generation by 15--25 percentage points on large models and surpassing the agentic approach despite the latter consuming 8\(\times\) more tokens. The benefit of structured intermediates is most pronounced for large models; for small models, schema compliance becomes the bottleneck. These findings suggest that in formally structured domains, constraining the LLM's output to a well-typed intermediate representation and delegating query construction to deterministic code yields better results than either unconstrained generation or iterative tool use.

Problem

• Large Language Models
• Static Analysis
• Natural Language Interface
• Query Generation
• Intermediate Representation

Innovation

• structured intermediate representation
• LLM delegation
• static analysis
• code query generation
• schema-constrained generation