A Matrix Logic Approach to Efficient Frequent Itemset Discovery in Large Data Sets

2024-12-27

AI Summary

To address the high computational and storage overhead of candidate generation and poor scalability in frequent itemset mining under big data settings, this paper proposes a novel mining algorithm grounded in Boolean matrix logic. Departing from conventional candidate-generation frameworks, the method directly computes itemset support counts via efficient AND/OR operations on Boolean transaction matrices. It further introduces a recursive matrix compression scheme coupled with an adaptive thresholding mechanism to significantly reduce both time and space complexity. As the first systematic work to apply Boolean matrix operations to frequent pattern mining, our approach demonstrates strong empirical performance on the Groceries dataset: it efficiently discovers large numbers of frequent itemsets at low support thresholds and precisely extracts strong association rules at high thresholds. Moreover, runtime and memory consumption scale gently with dataset size, confirming superior scalability and robustness.

Technology Category

• Data Mining & Knowledge Management: Rule Mining & Pattern Mining
• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Combinatorial Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

This paper proposes a frequent itemset mining algorithm based on the Boolean matrix method, aiming to solve the storage and computational bottlenecks of traditional frequent pattern mining algorithms in high-dimensional and large-scale transaction databases. By representing the itemsets in the transaction database as Boolean matrices, the algorithm uses Boolean logic operations such as AND and OR to efficiently calculate the support of the itemsets, avoiding the generation and storage of a large number of candidates itemsets in traditional algorithms. The algorithm recursively mines frequent itemsets through matrix operations and can flexibly adapt to different data scales and support thresholds. In the experiment, the public Groceries dataset was selected, and the running efficiency test and frequent itemset mining effect test were designed to evaluate the algorithm's performance indicators such as running time, memory usage, and number of frequent itemsets under different transaction numbers and support thresholds. The experimental results show that the algorithm can efficiently mine a large number of frequent itemsets when the support threshold is low, and focus on strong association rules with high support when the threshold is high. In addition, the changing trends of running time and memory usage show that the Boolean matrix method can still maintain good running efficiency when the number of transactions increases significantly and has high scalability and robustness. Future research can improve memory optimization and matrix block operations, and combine distributed computing and deep learning models to further enhance the algorithm's applicability and real-time processing capabilities in ultra-large-scale data environments. The algorithm has broad application potential and development prospects in the fields of market analysis, recommendation systems, and network security.

Problem

• Big Data
• Frequent Itemset Mining
• Efficiency

Innovation

• Boolean Matrix
• Efficient Data Mining
• Scalability