🤖 AI Summary
Current large language models (LLMs) are applied in linguistics in a fragmented, ad hoc manner, lacking systematic methodology and theoretical integration. To address this, we propose the first two-tiered, linguistics-oriented methodology framework, comprising three goal-aligned paradigms: prompt-based interaction, fine-tuned modeling, and embedding probing. The framework integrates prompt engineering, open-weight model fine-tuning (e.g., LLaMA), context-aware embedding quantification, and multi-stage research pipeline design. It prioritizes reproducibility, verifiability, and theory-driven inquiry. Empirical validation employs retrospective analysis, prospective experimentation, and expert surveys. Results demonstrate substantial improvements in methodological reproducibility and theoretical rigor, enabling linguistics to transition from empirical LLM application toward a robust, scientific paradigm grounded in computational and cognitive linguistic principles.
📝 Abstract
Large Language Models (LLMs) are transforming language sciences. However, their widespread deployment currently suffers from methodological fragmentation and a lack of systematic soundness. This study proposes two comprehensive methodological frameworks designed to guide the strategic and responsible application of LLMs in language sciences. The first method-selection framework defines and systematizes three distinct, complementary approaches, each linked to a specific research goal: (1) prompt-based interaction with general-use models for exploratory analysis and hypothesis generation; (2) fine-tuning of open-source models for confirmatory, theory-driven investigation and high-quality data generation; and (3) extraction of contextualized embeddings for further quantitative analysis and probing of model internal mechanisms. We detail the technical implementation and inherent trade-offs of each method, supported by empirical case studies. Based on the method-selection framework, the second systematic framework proposed provides constructed configurations that guide the practical implementation of multi-stage research pipelines based on these approaches. We then conducted a series of empirical experiments to validate our proposed framework, employing retrospective analysis, prospective application, and an expert evaluation survey. By enforcing the strategic alignment of research questions with the appropriate LLM methodology, the frameworks enable a critical paradigm shift in language science research. We believe that this system is fundamental for ensuring reproducibility, facilitating the critical evaluation of LLM mechanisms, and providing the structure necessary to move traditional linguistics from ad-hoc utility to verifiable, robust science.