🤖 AI Summary
This work addresses security risks—such as sensitive information leakage, unauthorized file access, and untrusted external network connections—arising from executing LLM-generated code within evaluation environments. We propose the first standardized sandbox security testing framework tailored for LLM evaluation infrastructures. Methodologically, we design multi-dimensional handcrafted adversarial test cases, integrate them into the open-source Dyff evaluation platform, and combine dynamic behavioral monitoring with privilege-boundary verification to quantitatively assess sandbox protection capabilities and guide hardening efforts. Our contributions are threefold: (1) the first systematic characterization of the boundary conditions under which LLMs generate malicious code under instruction-based prompting; (2) a reproducible mechanism for precise risk localization; and (3) empirically grounded evidence supporting effective mitigation strategies—collectively enhancing the robustness and trustworthiness of AI evaluation systems.
📝 Abstract
While large language models (LLMs) are powerful assistants in programming tasks, they may also produce malicious code. Testing LLM-generated code therefore poses significant risks to assessment infrastructure tasked with executing untrusted code. To address these risks, this work focuses on evaluating the security and confidentiality properties of test environments, reducing the risk that LLM-generated code may compromise the assessment infrastructure. We introduce SandboxEval, a test suite featuring manually crafted test cases that simulate real-world safety scenarios for LLM assessment environments in the context of untrusted code execution. The suite evaluates vulnerabilities to sensitive information exposure, filesystem manipulation, external communication, and other potentially dangerous operations in the course of assessment activity. We demonstrate the utility of SandboxEval by deploying it on an open-source implementation of Dyff, an established AI assessment framework used to evaluate the safety of LLMs at scale. We show, first, that the test suite accurately describes limitations placed on an LLM operating under instructions to generate malicious code. Second, we show that the test results provide valuable insights for developers seeking to harden assessment infrastructure and identify risks associated with LLM execution activities.