An innovative data collection method to eliminate the preprocessing phase in web usage mining

2023-04-01

Engineering Science and Technology, an International Journal

AI Summary

Conventional web usage mining relies on server logs, necessitating labor-intensive post-hoc preprocessing—including log parsing, cleaning, and session identification—which impedes real-time analysis and degrades analytical efficiency and accuracy. Method: We propose a “collection-as-preprocessing” paradigm that shifts preprocessing to the data generation source. Leveraging coordinated browser extensions and a lightweight proxy, we capture user interaction events in structured, semantically enriched form at the origin. Our approach integrates semantic session modeling with minimal HTTP transaction tagging to enable end-to-end analysis without downstream preprocessing. Contribution/Results: Evaluated on real-world websites, our method eliminates preprocessing latency entirely, achieves 99.2% session reconstruction accuracy, and accelerates downstream association rule mining by 5.8×. This work pioneers the systematic relocation of preprocessing to the data acquisition stage, significantly enhancing the timeliness, fidelity, and scalability of web usage mining.

Technology Category

• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Application Domains: Humanities & Computational Social Science
• Search and Optimization: Distributed Search

Application Category

• Web Mining and Content Analysis: Web data integration and cleaning
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Problem

• User Behavior Analysis
• Data Collection Methods
• Internet Usage

Innovation

• Data Collection Method
• Application Programming Interface
• Enhanced Data Analysis