SECite: Analyzing and Summarizing Citations in Software Engineering Literature

2026-01-12

AI Summary

This study addresses the limitations of traditional literature reviews, which rely heavily on authors’ self-reports and often fail to capture a paper’s actual impact and shortcomings within the scholarly community. To overcome this, the authors propose a novel approach that integrates citation sentiment analysis with generative artificial intelligence. Specifically, they employ natural language processing and unsupervised machine learning to classify the sentiment expressed in citation contexts of software engineering papers, then leverage these sentiment signals to guide large language models (LLMs) in generating sentiment-aware automatic summaries. This work represents the first integration of citation sentiment analysis with LLM-based summarization, successfully identifying strengths and weaknesses of nine target papers as perceived in external citations. The method reveals both alignments and discrepancies between community perspectives and author claims, offering a new paradigm for objectively assessing scholarly impact.

Technology Category

• Natural Language Processing: Summarization
• Data Mining & Knowledge Management: Data Visualization & Summarization
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Identifying the strengths and limitations of a research paper is a core component of any literature review. However, traditional summaries reflect only the authors'self-presented perspective. Analyzing how other researchers discuss and cite the paper can offer a deeper, more practical understanding of its contributions and shortcomings. In this research, we introduce SECite, a novel approach for evaluating scholarly impact through sentiment analysis of citation contexts. We develop a semi-automated pipeline to extract citations referencing nine research papers and apply advanced natural language processing (NLP) techniques with unsupervised machine learning to classify these citation statements as positive or negative. Beyond sentiment classification, we use generative AI to produce sentiment-specific summaries that capture the strengths and limitations of each target paper, derived both from clustered citation groups and from the full text. Our findings reveal meaningful patterns in how the academic community perceives these works, highlighting areas of alignment and divergence between external citation feedback and the authors'own presentation. By integrating citation sentiment analysis with LLM-based summarization, this study provides a comprehensive framework for assessing scholarly contributions.

Problem

• citation analysis
• sentiment analysis
• literature review
• scholarly impact
• software engineering

Innovation

• citation sentiment analysis
• generative AI
• unsupervised machine learning
• NLP
• scholarly impact assessment