Secure or Suspect? Investigating Package Hallucinations of Shell Command in Original and Quantized LLMs

2025-12-08

AI Summary

Quantization of large language models (LLMs) may exacerbate package hallucination and introduce vulnerable dependencies when generating Shell commands for Go package installation—risks with critical implications for software supply chain security. Method: We conduct the first systematic empirical study, evaluating five Qwen variants under full-precision, 8-bit, and 4-bit quantization across Stack Overflow, MBPP, and Paraphrase benchmarks. We measure both package hallucination rates and the frequency of known-vulnerable dependencies in generated installation commands. Results: Quantization significantly amplifies both risks: 4-bit models exhibit the highest hallucination rates, and vulnerability prevalence increases monotonically with reduced precision. Crucially, most hallucinated package names deliberately mimic legitimate GitHub or golang.org import paths—enhancing their deceptive realism. This work uncovers a previously underappreciated trade-off between model compression and supply-chain integrity, providing empirically grounded risk assessments and actionable guidance for secure, trustworthy LLM deployment in software development workflows.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models for code (LLMs4Code) are increasingly used to generate software artifacts, including library and package recommendations in languages such as Go. However, recent evidence shows that LLMs frequently hallucinate package names or generate dependencies containing known security vulnerabilities, posing significant risks to developers and downstream software supply chains. At the same time, quantization has become a widely adopted technique to reduce inference cost and enable deployment of LLMs on resource-constrained environments. Despite its popularity, little is known about how quantization affects the correctness and security of LLM-generated software dependencies while generating shell commands for package installation. In this work, we conduct the first systematic empirical study of the impact of quantization on package hallucination and vulnerability risks in LLM-generated Go packages. We evaluate five Qwen model sizes under full-precision, 8-bit, and 4-bit quantization across three datasets (SO, MBPP, and paraphrase). Our results show that quantization substantially increases the package hallucination rate (PHR), with 4-bit models exhibiting the most severe degradation. We further find that even among the correctly generated packages, the vulnerability presence rate (VPR) rises as precision decreases, indicating elevated security risk in lower-precision models. Finally, our analysis of hallucinated outputs reveals that most fabricated packages resemble realistic URL-based Go module paths, such as most commonly malformed or non-existent GitHub and golang.org repositories, highlighting a systematic pattern in how LLMs hallucinate dependencies. Overall, our findings provide actionable insights into the reliability and security implications of deploying quantized LLMs for code generation and dependency recommendation.

Problem

• Investigates quantization's impact on LLM-generated package hallucination rates
• Examines increased vulnerability presence in quantized models' dependency recommendations
• Analyzes patterns in hallucinated Go package names resembling real URLs

Innovation

• Evaluates quantization impact on LLM package hallucination rates
• Measures vulnerability presence in generated dependencies across precisions
• Analyzes hallucinated package patterns resembling real module paths