An AI-Powered Research Assistant in the Lab: A Practical Guide for Text Analysis Through Iterative Collaboration with LLMs

2025-05-14

AI Summary

Taxonomy construction for classifying unstructured text (e.g., personal goal statements) is time-intensive, prone to researcher bias, and suffers from poor reproducibility. Method: This paper proposes a human–AI collaborative, iterative text analysis paradigm that integrates top-down and bottom-up strategies to enable dynamic taxonomy generation, evaluation, refinement, and validation. Leveraging prompt engineering, it facilitates multi-turn collaboration between domain researchers and large language models (LLMs), with human feedback driving iterative taxonomy optimization. Intercoder reliability is quantified using Cohen’s κ within a structured coding framework. Results: Empirical evaluation in a life-domain dataset achieves κ > 0.85, significantly enhancing analytical efficiency, reliability, and reproducibility. This work pioneers deep integration of LLMs into the qualitative analysis closed loop, offering a novel methodology for low-bias, high-fidelity open-text classification.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Analyzing texts such as open-ended responses, headlines, or social media posts is a time- and labor-intensive process highly susceptible to bias. LLMs are promising tools for text analysis, using either a predefined (top-down) or a data-driven (bottom-up) taxonomy, without sacrificing quality. Here we present a step-by-step tutorial to efficiently develop, test, and apply taxonomies for analyzing unstructured data through an iterative and collaborative process between researchers and LLMs. Using personal goals provided by participants as an example, we demonstrate how to write prompts to review datasets and generate a taxonomy of life domains, evaluate and refine the taxonomy through prompt and direct modifications, test the taxonomy and assess intercoder agreements, and apply the taxonomy to categorize an entire dataset with high intercoder reliability. We discuss the possibilities and limitations of using LLMs for text analysis.

Problem

• Automating time-consuming text analysis to reduce bias
• Developing efficient taxonomies for unstructured data using LLMs
• Improving intercoder reliability in text categorization tasks

Innovation

• Iterative collaboration between researchers and LLMs
• Prompt-based taxonomy generation and refinement
• High intercoder reliability in dataset categorization