Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries

2026-07-18

AI Summary

This study addresses the inefficiency of manual curation in the Clinical and Translational Science Awards (CTSA) program, where compiling scholarly impact records requires approximately 15 hours per individual, hindering scalability. To overcome this limitation, the authors propose a novel human-in-the-loop AI agent that automatically aggregates multi-source data and generates scholar profiles grounded in the Translational Science Benefit Model (TSBM), complete with citation-backed evidence and a one-sentence impact summary for human review. The approach dramatically improves efficiency, reducing average review time to 14 minutes per person. Evaluators deemed 81.7% of the AI-generated content usable, with synthetic accuracy and utility scores reaching 4.5 and 4.8 out of 5, respectively. The system achieves recall comparable to manual methods while effectively capturing non-academic impact evidence often missed by traditional workflows.

Abstract

Introduction. Clinical and Translational Science Award (CTSA) programs must document their scholars' research impact, but assembling each scholar's record by hand takes staff an estimated 15 hours and does not scale to a full cohort. An artificial intelligence (AI) agent could serve as a tool to gather scholar data across platforms and disciplines. Methods. We built a human-in-the-loop AI agent that assembles a dossier of sourced evidence for each scholar and drafts one-sentence Translational Science Benefits Model (TSBM) impact summaries for staff review. We evaluated it in the impact-reporting workflow of one CTSA hub across 10 career-development (KL2/K12) scholars. Two evaluation staff independently coded all 507 findings as accept, edit, or reject; the primary measure was the unanimous usable rate, defined as the share both accepted or edited. Results. Both reviewers accepted or edited 81.7% of the agent's findings. Reviewers each spent a median of 14 minutes per scholar, replacing an estimated 15 hours of manual assembly. Inter-rater agreement was moderate (Cohen's kappa 0.43 on the usable-versus-reject decision). A profile discovery study found the agent's recall close to human search. The agent's impact evidence spanned all four TSBM domains, and about a third of the reviewed findings fell in non-scholarly categories that routine processes tend to miss. Reviewers rated synthesis accuracy 4.5 and usefulness 4.8 on a 5-point scale. Conclusions. A human-in-the-loop AI agent can serve as the first-pass author of a scholar's impact record, shifting staff from collecting and writing to reviewing, and making cohort-scale impact reporting feasible.

Problem

• research impact
• Clinical and Translational Science Award
• scholar evaluation
• impact reporting
• data aggregation

Innovation

• human-in-the-loop AI
• translational impact summary
• TSBM
• automated evidence synthesis
• scholar impact assessment