coder training

Although large language models (LLMs) can achieve agreement with human annotators in text coding, their judgments may rely on superficial features unrelated to the underlying theoretical construct, thereby lacking construct validity. To address this issue, this work proposes a “fine-grained calibration” approach that decomposes theoretical constructs into clause-level components, validates each component against extractive evidence, and aggregates results according to explicit theoretical rules to assess whether LLMs genuinely measure the target construct. This method shifts the validation of construct validity from output consistency to process interpretability, enabling identification of errors stemming either from missing components or confusion with neighboring constructs. It establishes a transparent and interpretable paradigm for trustworthy measurement using LLMs in the social sciences.

As AI coding agents become increasingly prevalent in scientific research, the long-standing assumption that “at least one human understands the intent behind every piece of code” is being undermined, jeopardizing code readability and collaborative maintenance. This study investigates this emerging challenge through contextual inquiry, a survey of over 800 research software developers, case studies, and empirical analysis of version control logs. We introduce the concept of “landmarking strategies,” revealing how researchers explicitly demarcate human-readable code from agent-specific context. Our findings also uncover non-standard uses of shared infrastructure such as Git, highlighting risks of destandardization in current practices. The work offers critical design implications for improving the maintainability of scientific codebases and enhancing team collaboration in agent-augmented research environments.

This study addresses a critical gap in machine learning education: the overreliance on pre-labeled datasets, which often obscures the subjectivity and ambiguity inherent in data annotation, leading students to place undue trust in model outputs. To counter this, the authors introduce an innovative pedagogical intervention that transforms manual annotation into an active learning tool. Students annotated hair coverage in skin lesion images using a three-point scale, followed by structured reflections via questionnaires. A cross-institutional experiment involving 43 participants from Fontys University of Applied Sciences (Netherlands) and the IT University of Copenhagen (Denmark) demonstrated that this approach significantly enhanced learners’ awareness of annotation ambiguity, dataset biases, and model limitations. Most participants acknowledged the influence of personal interpretation on labeling decisions and reported higher engagement compared to traditional instruction. This work provides the first empirical evidence supporting subjective annotation as an effective strategy for cultivating critical thinking about AI systems.

Qualitative evaluation of inductive coding faces significant challenges: conventional metrics are ill-suited for exploratory processes; manual assessment is labor-intensive; and expert-annotated “ground truth” introduces methodological limitations. This paper introduces the first quantifiable evaluation framework for open coding in grounded theory and thematic analysis. Moving beyond the conventional human–AI alignment paradigm, it innovatively integrates stability assessment (Cohen’s kappa, Jaccard similarity) with cross-human–machine comparison as a dual-validation mechanism. Crucially, the framework operates without presupposing ground-truth labels, enabling bias detection and coding quality measurement in human–AI collaborative workflows. Evaluated on two HCI datasets, the framework demonstrates high inter-coder agreement (κ > 0.75), strong output stability (Jaccard similarity > 0.89 across repeated runs), and yields a reusable, AI-augmented coding workflow.

Existing evaluation methods for LLM-generated code comments rely on small-scale datasets and inadequate IR metrics (e.g., BLEU), failing to capture semantic fidelity. Method: We systematically assess GPT-3.5’s Javadoc generation for 23,850 Java code snippets, employing a dual-dimensional evaluation combining quantitative BLEU scoring with qualitative expert human assessment. Contribution/Results: Our study reveals a critical flaw in BLEU: high scores frequently correlate with low-quality, verbatim descriptions, while high-fidelity semantic paraphrasing is systematically penalized. We find that 69.7% of generated Javadocs are semantically equivalent to—or can be refined to match—the original quality, and 22.4% significantly surpass the originals. These results demonstrate that automated metrics alone are unreliable for assessing documentation quality. We advocate human evaluation as the gold standard, with BLEU serving only as a supplementary heuristic—establishing a new, more rigorous paradigm for evaluating code documentation generation.

While large language models (LLMs) can achieve high accuracy in coding political events, they often fail to faithfully adhere to expert-defined coding rules, leading to unreliable behavior. This study systematically evaluates LLMs’ logical consistency under controlled perturbations—such as variations in label names and coding order—by enhancing structured codebooks through precise terminology, illustrative examples, retrieval-augmented context, and rules for challenging cases, combined with prompt engineering. The work reveals, for the first time, a critical disconnect between predictive performance and behavioral reliability, demonstrating that high accuracy does not necessarily imply compliance with social science coding logic. Although the refined codebooks substantially improve fine-grained classification performance, the models remain sensitive to minor codebook modifications, underscoring the necessity of explicit reliability assessment in computational social science applications.

This study addresses the fragmented understanding of “vibe coding” by systematically synthesizing knowledge dispersed across academic and practitioner literature. Employing a unified protocol and a multi-voiced literature review methodology, it analyzes 47 peer-reviewed and gray literature sources from 2022 to 2025, revealing that vibe coding is fundamentally an intent-driven iterative cycle of generation, evaluation, and refinement. The findings indicate that 45% of the reviewed works report short-term productivity gains, with the strongest empirical support for its efficacy in prototyping and UI development. However, significant evidence gaps persist regarding its applicability in production-grade, data-intensive, and safety-critical contexts. This work provides a structured empirical foundation for understanding the evolving role of developers and delineating the appropriate boundaries for vibe coding adoption.

Traditional manual code review struggles to meet the high-throughput, low-cost demands of AI-assisted software development and faces scalability limitations in human-in-the-loop workflows. This work proposes a novel paradigm that fully replaces human reviewers with a large language model–driven autonomous coding agent, integrating capabilities in code comprehension, generation, testing, and repair within a unified framework. Experimental results demonstrate that the proposed agent not only fulfills all conventional code review objectives but also significantly outperforms existing approaches in terms of cost, efficiency, and scalability. These findings suggest the potential to fundamentally reshape software quality assurance practices that have remained largely unchanged for over five decades.

This study addresses the inefficiencies in collaborative code review caused by mismatches in how developers interpret feedback. To bridge this gap, it introduces a personality-aware mechanism and employs a mixed-methods user study—combining persona modeling and qualitative analysis—to investigate personalized review feedback tailored to developers’ problem-solving styles, experience levels, and roles. Findings reveal that developers prefer feedback offering explanatory depth, learning support, actionable suggestions, and risk awareness over mere conciseness. By striking a balance between personalization and clarity with credibility, this work advances the development of human-centered AI-assisted code review systems.

This study investigates how users with varying levels of programming experience—non-programmers, novices, and professional developers—differ in their practices of vibe coding, a paradigm wherein code is generated via natural language prompts and validated through execution. Drawing on a mixed-methods survey of 162 participants, the research integrates quantitative and qualitative analyses to reveal distinct patterns in motivation, interaction strategies, and quality assurance behaviors: non-programmers prioritize accessibility, novices emphasize learning, and professionals seek efficiency. The work introduces the concept of a “perception–action gap,” highlighting that while users broadly recognize the risks associated with AI-generated code, their actual ability to verify its correctness remains heavily contingent on programming expertise. This finding underscores the limitations of vibe coding in achieving true code democratization.