capability gap analysis

This work proposes a systematic approach to derive task effectiveness requirements in the absence of explicit user needs. The method deconstructs task intent into context, functionality, constraints, critical dimensions, performance attributes, and architectural solutions, and introduces a task complexity factor to quantify the impact of external challenges and technology maturity. By integrating Best-Worst Scaling, it prioritizes critical dimensions based on stakeholder judgments. Through task decomposition modeling and quantitative complexity analysis, the framework supports integration with UAF/SysML artifacts and establishes a traceable mechanism for generating Tier 1 and Tier 2 requirements. The approach is validated using a close air support mission case study, effectively addressing a critical gap in requirements engineering when clear initial inputs are unavailable.

This study addresses the lack of systematic preprocessing standards, integrated analytical workflows, and cross-method consistency checks in current computer-based assessment process data. To bridge this gap, the authors propose an end-to-end analytical framework featuring a unified preprocessing pipeline and a dual-path analysis paradigm that synergistically combines feature engineering with model-based inference. The framework incorporates large language models (LLMs) to standardize action sequences and facilitate process-data-driven differential item functioning (DIF) detection. Technically, it integrates timestamp correction, action chunking, n-gram and TF-IDF feature extraction, multidimensional scaling, hidden Markov modeling, and subtask identification. Empirical results demonstrate that n-gram–based behavioral clustering offers diagnostic value for incorrect responders, multidimensional scaling effectively reconstructs behavioral constructs, and process data can identify and mitigate construct-irrelevant group differences.

The proliferation of generative AI has rendered traditional modular assessments ineffective, creating a critical misalignment between academic evaluation and industry practices augmented by AI. Method: This paper proposes an AI-resilient assessment framework for computing education, centered on interlinked, multi-stage problem structures with output feedback loops—designed to deter AI-assisted cheating and bridge the pedagogy–practice gap. Contribution/Results: We formally prove that interlinked problems exhibit superior AI resilience compared to modular ones. Empirical validation across four data science courses (N=138) demonstrates that semi-structured interlinked tasks yield more stable and valid measures of student competency than open-ended projects—challenging the prevailing policy assumption that openness inherently deters AI misuse. Under AI access, modular assignments achieved near-perfect scores, while proctored exam performance dropped by 30%; in contrast, interlinked projects maintained high validity (r > 0.7) and significantly curbed AI substitution, thereby preserving academic integrity and assessment fidelity.

Existing automated data unit testing methods often overlook the semantic requirements imposed by downstream tasks, limiting their ability to ensure data reliability. This work proposes a task-aware data testing generation mechanism that statically analyzes task code and data characteristics to identify data access patterns and infer implicit assumptions, thereby automatically generating executable, task-oriented tests. We introduce the SIFTA framework, which integrates large language model prompting with sparse execution feedback to dynamically refine test prompts. Evaluated on a new benchmark encompassing five datasets and sixty tasks, SIFTA significantly outperforms both task-agnostic and task-specific baselines, with its automatically generated prompts surpassing those crafted manually or produced by general-purpose optimizers.

This work addresses the complex challenges of continuously monitoring item pool quality and health in large-scale AI-driven assessments. It proposes AQuAP, a dashboard system integrated with an item factory framework that leverages operational data analytics to support item generation and pool management. The system introduces novel metrics such as Effective Bank Size (EBS), which combines exposure rates and usage frequency to holistically evaluate the security, diversity, and efficiency of the item pool. By integrating psychometric indicators, exposure control algorithms, and advanced visualization techniques, AQuAP enables real-time monitoring of item pool vitality. The system has been successfully deployed in the Duolingo English Test, significantly enhancing the intelligence and responsiveness of item pool management.

Current evaluation methods for large language models (LLMs) primarily identify failing samples or categories but struggle to uncover underlying capability deficiencies, thereby limiting targeted model improvement. This work proposes CRAFT, a novel framework that diagnoses model weaknesses at the scoring-criterion level. CRAFT constructs a hierarchical capability tree by extracting capability descriptions and applying hierarchical clustering, then dynamically identifies low-performance nodes across multiple granularities to generate targeted fine-tuning data. Evaluated on financial and legal domains as well as 13 standard benchmarks, CRAFT significantly outperforms prompt-clustering and random data generation baselines. Fine-tuning four open-source LLMs with CRAFT-generated data consistently enhances their performance, demonstrating more precise localization of capability gaps and enabling efficient, targeted model refinement.

This work addresses the challenge of balancing software quality, testability, and maintainability under rapid iteration and frequent requirement changes. It proposes Algorithm-Driven Development (ADD), a novel approach that unifies requirements specification and technical design by using algorithm flowcharts as a single, coherent artifact. This integration enables end-to-end modeling of requirements, architecture, and testing. Leveraging this model, the system automatically generates high-coverage acceptance tests and incorporates continuous integration with code coverage feedback. Industrial adoption at Dassault Systèmes demonstrates that ADD achieves over 95% code coverage, substantially reduces defect density, and ensures a stable delivery cadence, outperforming conventional test-driven development and test-after approaches.

Traditional AI evaluation methods are primarily designed for static model selection and often fail to diagnose root causes of performance degradation in production or guide targeted improvements. This work proposes EvalLoop, a novel methodology that embeds evaluation into a continuous optimization loop. By integrating dimensional metric grouping, failure mode categorization, single-variable controlled experiments, and a human-in-the-loop gating mechanism, EvalLoop enables precise mapping from failure attribution to actionable refinements and supports deployment-aware model selection. Evaluated on a sales intelligence briefing generation task, the approach increased overall accuracy of the best-performing model from 82.6% to 94.6%, improved performance on critical dimensions by over 16 percentage points, and reduced human review effort by 94%.

This work addresses a critical limitation in existing cloud skill testing, which focuses solely on task success rates and fails to reveal uncovered behaviors, leaving test adequacy unquantifiable. To bridge this gap, the study introduces the first formal definition of test coverage units, coverage relationships, and a computational pipeline for cloud skills. It proposes a coverage evaluation method grounded in natural language skill packages: by parsing user prompts and initial resource states, the approach reconstructs operational obligations and models workflow context to establish an end-to-end measurement pipeline. Integrating model-assisted candidate generation with expert review, the framework produces auditable coverage reports and closes the loop by mapping coverage gaps to source-level test improvement recommendations. Empirical results demonstrate that this methodology enables quantitative assessment of test coverage for production-grade cloud skills, substantially enhancing their reliability and observability.