Overview of ExpertLifeCLEF 2018: how far automated identification systems are from the best experts?

2025-09-25

AI Summary

Quantifying human expert uncertainty in species identification and systematically benchmarking deep learning models against human performance. Method: We conducted the first rigorous, controlled comparison between 19 state-of-the-art multimodal (vision + audio) deep learning systems and nine leading French botanists on a large-scale, expert-annotated biodiversity dataset—representing the first such evaluation in real-world biological identification. Contribution/Results: The top-performing AI models achieve accuracy comparable to, and in certain taxonomic groups exceeding, human experts; critically, their predictive uncertainty distributions closely align with those of human experts. This work establishes a standardized, human–machine comparable evaluation framework and provides the first empirical evidence that AI systems possess tangible potential to either augment or replace domain experts in biodiversity monitoring applications.

Technology Category

• Machine Learning: Multimodal Learning
• Humans and AI: Voting
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Automated identification of plants and animals has improved considerably in the last few years, in particular thanks to the recent advances in deep learning. The next big question is how far such automated systems are from the human expertise. Indeed, even the best experts are sometimes confused and/or disagree between each others when validating visual or audio observations of living organism. A picture actually contains only a partial information that is usually not sufficient to determine the right species with certainty. Quantifying this uncertainty and comparing it to the performance of automated systems is of high interest for both computer scientists and expert naturalists. The LifeCLEF 2018 ExpertCLEF challenge presented in this paper was designed to allow this comparison between human experts and automated systems. In total, 19 deep-learning systems implemented by 4 different research teams were evaluated with regard to 9 expert botanists of the French flora. The main outcome of this work is that the performance of state-of-the-art deep learning models is now close to the most advanced human expertise. This paper presents more precisely the resources and assessments of the challenge, summarizes the approaches and systems employed by the participating research groups, and provides an analysis of the main outcomes.

Problem

• Evaluating automated species identification against human expertise
• Quantifying uncertainty in visual organism identification accuracy
• Comparing deep learning systems with expert botanists' performance

Innovation

• Deep learning systems compared with expert botanists
• Automated identification close to human expertise performance
• Evaluation of 19 models against 9 French flora experts