Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA

2026-07-21

AI Summary

This study addresses the high fine-tuning costs and associated risks—such as knowledge degradation, diminished instruction-following capability, and increased hallucination—faced by small-scale large language models in cybersecurity question-answering tasks. To mitigate these issues, the authors propose the FiT diagnostic framework, which evaluates model suitability prior to fine-tuning along three dimensions: lexical recognition, parametric knowledge, and contextualization of retrieved information. The work establishes the first task-oriented diagnostic system tailored for cybersecurity QA, integrating knowledge-focused and instruction-focused fine-tuning paradigms with retrieval-augmented evaluation. Through empirical analysis of five open-source 7B models, the study reveals that fine-tuning generally impairs lexical and parametric knowledge: knowledge-focused fine-tuning induces mild yet consistently ranked degradation, whereas instruction-focused fine-tuning triggers severe knowledge collapse while preserving the ability to contextualize retrieved information. Notably, FiT scores effectively predict post-fine-tuning performance trends.

Abstract

Large Language Models (LLMs) are increasingly fine-tuned for critical-domain Question-Answering (QA), yet choosing which small model to adapt, before paying the cost of adaptation, remains difficult. Fine-tuning can improve domain alignment, but it may also erode prior knowledge, weaken instruction-following, or increase hallucination, especially when labeled data are scarce or rapidly evolving as in cybersecurity. We present FiT (Find before Fine-Tune), a task-oriented diagnostic framework that characterizes small LLMs along three capabilities required for cybersecurity QA: vocabulary recognition, parametric knowledge, and contextualization of retrieved information. Using FiT, we conduct an empirical study of five open-weight 7-billion-parameter models under two fine-tuning regimes. We find that fine-tuning does not uniformly help: it consistently degrades vocabulary and parametric knowledge in small models, and the two regimes trade off differently. Knowledge-focused tuning causes moderate, rank-preserving degradation, whereas instruction-focused tuning collapses measured knowledge through induced abstention, inverting the knowledge ranking while leaving retrieval-grounded contextualization essentially intact. We quantify these regime-specific patterns with rank-correlation analysis and show that pre-fine-tuning FiT scores anticipate the direction of post-tuning change. Our results suggest that task-oriented diagnosis can screen out unsuitable models, avoid unnecessary fine-tuning, and support safer deployment of small LLMs in cybersecurity QA pipelines.

Problem

• small LLMs
• cybersecurity QA
• fine-tuning
• model selection
• diagnostic evaluation

Innovation

• diagnostic framework
• small LLMs
• cybersecurity QA
• fine-tuning regimes
• knowledge degradation