Problem
Research questions and friction points this paper is trying to address.
Network Threat Intelligence
Efficiency
Information Source Discovery
Bandit on the Hunt: Dynamic Crawling for Cyber Threat Intelligence
🤖 AI Summary
Current cyber threat intelligence (CTI) collection faces challenges including high uncertainty in information sources, low efficiency in discovering novel sources, and time-inefficient crawling. To address these issues, this paper proposes a dynamic focused crawler framework based on the Multi-Armed Bandit (MAB) paradigm—the first application of MAB to CTI acquisition. The framework enables adaptive seed-source expansion and online optimization of crawling paths. It integrates SBERT-based semantic matching, topic-focused webpage classification, and incremental seed updating to accurately identify high-value unstructured threat sources—such as threat overview pages, datasets, and malicious domains. Experimental results demonstrate that the approach achieves an information harvest rate exceeding 25%, expands the seed-source pool by over 300%, and significantly improves both coverage breadth and timeliness of CTI while maintaining high topical focus.
📝 Abstract
Public information contains valuable Cyber Threat Intelligence (CTI) that is used to prevent attacks in the future. Ideally, the learnings from previous attacks help to mitigate all those that follow. While there are standards for sharing this information, much of it is shared in non-standardized news articles or blog posts. It is a time-consuming task to monitor online sources for threats and even then, one can never be sure, to use the right sources. Current research propose extractors of Indicators of Compromise from known sources, while the identification of new sources is rarely considered. This paper proposes a focused crawler focused on the CTI domain based on multi-armed bandit ( MAB) and different crawling strategies. It uses SBERT to identify relevant documents, while dynamically adapt its crawling path. We propose a system called ThreatCrawl, which achieve a harvest rate of over 25% and is able to expand its used seed by over 300%, while retaining focus on the topic at hand. In addition, this crawler identified previously unknown but highly relevant overview pages, datasets, and domains.
Innovation
Methods, ideas, or system contributions that make the work stand out.
ThreatCrawl
Multi-Armed Bandit Algorithm
SBERT
🔎 Similar Papers
No similar papers found.
