🤖 AI Summary
This study addresses the lack of standardized, dynamic-aware approaches for assessing FAIR (Findable, Accessible, Interoperable, Reusable) compliance in geospatial datasets, particularly concerning evolving content and dataset-specific identifiers. To this end, the authors propose the first multi-agent collaborative framework for FAIR evaluation, integrating structured metadata extraction with 13 large language model–based evaluators, each aligned with a specific FAIR sub-principle. A critic agent is introduced to verify the quality and consistency of supporting evidence, enabling fine-grained scoring, automated evidence citation, and closed-loop reassessment. Experiments on 50 datasets yield average scores of 79.7%, 70.4%, 45.3%, and 72.0% across the four FAIR dimensions. Incorporating the critic agent improves inter-evaluator consistency by 18% to 89%, achieving a Fleiss’ kappa of 0.71, at a cost of approximately $0.054 per evaluation, thereby significantly enhancing transparency, consistency, and auditability.
📝 Abstract
Geospatial datasets support applications from urban planning to climate modeling, yet consistent assessment of FAIR compliance is difficult. Existing evaluators use different rubrics and evidence sources and may fail on JavaScript-rendered pages or repository-specific identifiers. For 50 datasets from 10 repositories, the standard deviation of normalized scores across available tools averages 15.0 percentage points and reaches 30.3 for one dataset. Because these outputs are not equivalent measurements, we use them to characterize disagreement and failure modes, not comparative accuracy. We present AgentFAIR, a multi-agent framework combining structured metadata extraction with 13 sub-principle-specific LLM evaluators. Each produces a 0-3 maturity score, cited evidence, and recommendations; a critic checks evidence and consistency and can request targeted re-evaluation. Mean Findability, Accessibility, Interoperability, and Reusability scores are 79.7%, 70.4%, 45.3%, and 72.0%. Rank correlations with four baseline tools range from 0.31 to 0.61; the FAIR-enough comparison is not statistically significant. On a 10-dataset repeated-run subset, sub-principle agreement averages 89% (standard deviation: 3 percentage points), versus 71% without the critic. A preliminary 15-dataset expert study yields Fleiss' kappa of 0.71 and 82% alignment with expert consensus. API cost is approximately USD 0.054 per dataset. These results support auditability and feasibility, while the limited benchmark, incomplete ablations, and single-model-family validation constrain claims about accuracy and generalization.