SBASH: a Framework for Designing and Evaluating RAG vs. Prompt-Tuned LLM Honeypots

2025-10-24

AI Summary

Honeypot effectiveness hinges on sustained attacker engagement, yet context-awareness—enabling dynamic adaptation to novel attacks, system states, and adversarial behavior—remains a critical bottleneck. Cloud-based large language models (LLMs) incur prohibitive latency, operational cost, and data privacy risks. This paper introduces SBASH: a lightweight, on-device LLM framework integrating retrieval-augmented generation (RAG) and system-level prompt tuning to enhance interaction authenticity while ensuring strict data confinement. A key finding from controlled experiments is that carefully engineered system prompts achieve RAG-level accuracy with substantially reduced inference latency. Authenticity is rigorously validated via similarity metrics—including Levenshtein distance, SBERT, and BERTScore. To our knowledge, this is the first systematic study demonstrating that prompt engineering can effectively substitute for RAG in local LLM honeypots, thereby simultaneously improving security, real-time responsiveness, and deployment efficiency.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Honeypots are decoy systems used for gathering valuable threat intelligence or diverting attackers away from production systems. Maximising attacker engagement is essential to their utility. However research has highlighted that context-awareness, such as the ability to respond to new attack types, systems and attacker agents, is necessary to increase engagement. Large Language Models (LLMs) have been shown as one approach to increase context awareness but suffer from several challenges including accuracy and timeliness of response time, high operational costs and data-protection issues due to cloud deployment. We propose the System-Based Attention Shell Honeypot (SBASH) framework which manages data-protection issues through the use of lightweight local LLMs. We investigate the use of Retrieval Augmented Generation (RAG) supported LLMs and non-RAG LLMs for Linux shell commands and evaluate them using several different metrics such as response time differences, realism from human testers, and similarity to a real system calculated with Levenshtein distance, SBert, and BertScore. We show that RAG improves accuracy for untuned models while models that have been tuned via a system prompt that tells the LLM to respond like a Linux system achieve without RAG a similar accuracy as untuned with RAG, while having a slightly lower latency.

Problem

• Evaluating RAG vs prompt-tuned LLMs for honeypot context-awareness improvement
• Addressing LLM challenges in accuracy, latency and data protection for honeypots
• Developing framework to enhance attacker engagement through realistic system responses

Innovation

• Lightweight local LLMs address data-protection issues
• RAG improves accuracy for untuned LLM models
• Prompt-tuned LLMs achieve similar accuracy without RAG