HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems

2026-08-01

AI Summary

This study addresses the challenges of inconsistent and costly human relevance annotations in content discovery systems, as well as the lack of effective coordination mechanisms in existing large language model–based evaluation approaches. To this end, the authors propose a hierarchical multi-agent relevance assessment framework comprising four specialized agents—query analyzer, item analyzer, relationship analyzer, and relevance judge—that operate through a layered coordination mechanism to dynamically orchestrate expert analyses and integrate external knowledge. The framework introduces, for the first time, a hierarchical coordination structure to enable orderly multi-agent collaboration and demonstrates its critical contribution to performance gains. Experimental results show that the proposed method significantly outperforms eleven baselines across five datasets, achieving a maximum relative improvement of 38% in Expected Valuation Score (EVS), with the hierarchical coordination mechanism alone contributing a 12.7% gain over uncoordinated collaboration.

Abstract

Content discovery systems depend on relevance judgment for search quality evaluation, but human annotation faces inter-annotator disagreement and scaling costs. While Large Language Models show promise as automated assessors, current approaches rely on flat aggregation strategies: single-step prompting, voting ensembles, or uncoordinated multi-agent pipelines that aggregate independent outputs without integration. We propose HIERA, a hierarchical multi-agent relevance assessment framework with four specialized agents: a Relevance Judge, Query Analyzer, Item Analyzer, and Relation Analyzer. The Judge determines when specialist analysis is needed; the Relation Analyzer then coordinates query and item analyses with external knowledge to establish relevance relationships for final judgment. Ablation studies show that the same agents and external knowledge without hierarchical coordination degrade performance, confirming that the coordination structure itself accounts for the improvement. Evaluation across five datasets (EVS, MSRD, ESCI, WANDS, Home Depot) shows improvements over 11 baselines: 10.2\% on Home Depot, 4.8\% on ESCI, and up to 38\% on EVS ($p < 0.05$). Hierarchical coordination yields 12.7\% improvement over uncoordinated collaboration using identical agents.

Problem

• relevance assessment
• content discovery systems
• multi-agent systems
• large language models
• annotation scalability

Innovation

• hierarchical multi-agent
• relevance assessment
• content discovery
• LLM-based evaluation
• agent coordination