llm cross-checking

This work addresses the critical issue that explanations generated by large language models (LLMs) often diverge from their underlying reasoning, undermining trustworthiness and transparency. To tackle this, we propose a collaborative consistency diagnosis framework that jointly leverages rule-based symbolic information extraction—identifying entities and logical structures—and fine-tuned LMs to automatically generate highly relevant, diverse, and targeted follow-up questions. Crucially, our approach establishes the first organic synergy between symbolic systems and LLMs in follow-up question generation, markedly improving detection of internal contradictions and critical omissions within explanations. Evaluated across multiple explanation evaluation benchmarks, our method achieves a 27.4% absolute gain in inconsistent explanation identification accuracy over strong LLM-only baselines and existing detection techniques. This work introduces an efficient, scalable, and principled paradigm for consistency assessment in explainable AI.

Existing LLM-based automatic evaluation methods rely on predefined, generic criteria, limiting generalization to unseen instructions and exhibiting insufficient robustness in quantitative and structural constraint assessment. To address these limitations, we propose ARJudge—a novel “Analysis–Refinement” framework. The Analyzer module fine-tunes an open-source LLM to adaptively generate multidimensional evaluation criteria; the Refiner module performs zero-shot joint discrimination of numerical accuracy, format compliance, and other structural constraints by integrating semantic analysis with code execution verification. ARJudge is trained on a composite corpus covering three task categories: criterion generation, textual analysis, and code analysis. Extensive experiments demonstrate that ARJudge significantly outperforms state-of-the-art fine-tuned evaluators across multiple benchmarks, particularly improving generalization to unseen instructions and enhancing discrimination accuracy for numerical precision and syntactic/format constraints.

Competency modeling is widely used in human resource management to select, develop, and evaluate talent. However, traditional expert-driven approaches rely heavily on manual analysis of large volumes of interview transcripts, making them costly and prone to randomness, ambiguity, and limited reproducibility. This study proposes a new competency modeling process built on large language models (LLMs). Instead of merely automating isolated steps, we reconstruct the workflow by decomposing expert practices into structured computational components. Specifically, we leverage LLMs to extract behavioral and psychological descriptions from raw textual data and map them to predefined competency libraries through embedding-based similarity. We further introduce a learnable parameter that adaptively integrates different information sources, enabling the model to determine the relative importance of behavioral and psychological signals. To address the long-standing challenge of validation, we develop an offline evaluation procedure that allows systematic model selection without requiring additional large-scale data collection. Empirical results from a real-world implementation in a software outsourcing company demonstrate strong predictive validity, cross-library consistency, and structural robustness. Overall, our framework transforms competency modeling from a largely qualitative and expert-dependent practice into a transparent, data-driven, and evaluable analytical process.

This paper addresses the prevalent issues of over-reliance and over-confidence in large language models (LLMs) during tool invocation. To recalibrate models’ awareness of their knowledge boundaries, we propose a multi-objective alignment framework. Methodologically, it introduces: (1) a novel knowledge boundary estimation technique grounded in consistency checking and absolute confidence scoring; and (2) a dynamic decision integration mechanism that jointly leverages probabilistic modeling, supervised fine-tuning, and inference-time intervention. Extensive experiments across diverse scenarios demonstrate that our approach significantly reduces redundant tool calls by 37.2% on average, while preserving task performance. Moreover, it improves response latency and lowers computational cost—achieving, for the first time, simultaneous optimization of reliability, efficiency, and cost-effectiveness in tool-augmented LLMs.

This work addresses the limitations of existing evaluation methods for assessing large language models’ ability to use external tools in complex real-world scenarios, which often suffer from oversimplified toolsets, rigid workflows, or subjective scoring. To this end, we present the first large-scale benchmark grounded in real Model Context Protocol (MCP) servers, encompassing 36 MCP services, 220 tools, and 1,000 multi-step natural language tasks that require agents to autonomously discover and orchestrate multiple tools. The evaluation employs a no-tool-name prompting strategy and a fine-grained, fact-based scoring mechanism, supported by a containerized framework and multidimensional diagnostic metrics—including tool discovery, parameterization, and error recovery. Experiments reveal that state-of-the-art models achieve pass rates exceeding 50%, with primary failure modes stemming from insufficient tool utilization and task comprehension errors. The benchmark framework, task schema, and a public subset of 500 tasks are openly released.

This study addresses key challenges faced by social science researchers when using large language models (LLMs) for text annotation—namely, poor reproducibility, annotation errors that compromise statistical inference, and high technical barriers. To overcome these issues, the authors propose the first end-to-end LLM-based text annotation framework tailored specifically for the social sciences and humanities (SSH). The framework integrates structured prompt engineering, open-source LLM API integration, cross-validation, and error propagation modeling, with an explicit emphasis on avoiding prompt overfitting and quantifying annotation uncertainty. Implemented in both Python and R, this approach establishes a transparent, reproducible, and scalable workflow that substantially enhances the reliability, efficiency, and methodological rigor of automated text annotation in SSH research.

This work addresses the limitations of existing automatic evaluation methods for large language model (LLM) outputs, which often rely on reference texts and exhibit limited generalizability, thereby struggling to accurately assess the quality and relevance of generated content. The authors propose a reference-free, domain-agnostic automated evaluation framework that leverages pairwise comparisons among multiple LLMs, integrated with an Elo rating system to produce stable and interpretable rankings. A tunable consistency threshold is introduced to balance evaluation confidence against coverage. Evaluated on scientific abstract quality assessment, the method yields rankings that align closely with expert judgments, significantly reducing the need for manual evaluation while demonstrating near-expert assessment capability.

This study addresses the current lack of interdisciplinary understanding regarding the integration pathways, efficacy boundaries, and systemic risks of large language models (LLMs) across natural sciences, social sciences, and humanities. Through a systematic literature review and illustrative case analyses, it critically evaluates the deployment of LLMs throughout the research lifecycle—including hypothesis generation, literature synthesis, data analysis, and scholarly writing. The work identifies ten previously underappreciated systemic risks, such as diminished researcher autonomy, AI-induced confirmation bias, ambiguous authorship, and inequitable access to technology. It further demonstrates how LLMs, while enhancing efficiency, simultaneously introduce challenges like hallucination, irreproducibility, data bias, and model opacity. To guide responsible adoption, the study proposes an interdisciplinary governance framework and a roadmap for explainable AI research in scholarly contexts.

This study addresses the issue of anthropomorphic outputs in large language models (LLMs) within software development toolchains, which often lead users to misattribute intentionality and comprehension capabilities, thereby undermining verification behaviors and disrupting trust calibration. The authors propose the first systematic, deployable set of output-side linguistic rules—comprising seven principles—implemented via a configuration-based system prompt that constrains known anthropomorphizing mechanisms without requiring any model modifications. Evaluated using the AnthroScore metric across 780 dialogue rounds, this approach significantly reduces anthropomorphic expressions (97% fewer anthropomorphic markers; AnthroScore reduction of 1.94 versus 0.96 in controls, p<0.001) while simultaneously shortening output length by 49%, effectively clarifying the machine’s non-human identity.

This work addresses the lack of reproducible and calibratable prompt engineering pipelines for evidence synthesis tasks in current large language models (LLMs). It proposes an innovative workflow that decouples scientific task specifications from prompting frameworks for the first time, leveraging annotated data and explicit metrics to drive prompt optimization. The approach operationalizes the entire pipeline into artifacts using the DSPy and GEPA toolchains, employing a small student model to execute tasks while a larger reflection model guides iterative refinement. Supporting structured task definitions, metric-driven search, and cross-framework portability, the method demonstrates strong compilability and artifact consistency in title and abstract screening tasks. Empirical validation further reveals the critical impact of optimization budgets on small-model performance, significantly enhancing the reliability and transparency of LLM-based applications.