MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models

2026-08-03

AI Summary

Current evaluation methods for large language models struggle to effectively link user interaction trajectories with their custom task objectives and actual outcomes, lacking validity feedback from real-world scenarios. This work proposes the first open-source, community-driven evaluation framework that unifies user-defined task goals, full dialogue trajectories, and outcome assessment. By employing a structured protocol to integrate conversation logs with user feedback, the framework enables large-scale participatory evaluation. A two-week pilot study collected 206 complete evaluation reports, revealing an average user satisfaction score of 4.19 out of 5, yet a task failure rate as high as 23.1%. Notably, multi-turn dialogues exhibited a failure rate 2.5 times that of single-turn interactions, underscoring significant deficiencies in current models when handling complex tasks.

Abstract

Benchmark suites assess model capability on controlled tasks; large-scale conversation corpora capture naturalistic use without user feedback; and in-interface feedback mechanisms record satisfaction without task purpose. Together, they leave a critical gap in LLM evaluation: no existing infrastructure routinely links interaction trajectories to user-defined outcomes. We introduce MonitrLLM, open-source infrastructure for community-centered LLM evaluations that links full conversation transcripts to user-reported task intent and outcome assessments, treating all three as primary evaluative signals rather than optional metadata. To demonstrate the value of this approach, we conducted a two-week feasibility pilot with 26 college students using ChatGPT, collecting 206 evaluation reports with full conversation transcripts. The findings from our pilot demonstrate the value of connecting conversation trajectories with user-reported outcomes. For instance, despite reporting high average satisfaction (4.19/5) with their LLM interactions, participants also experience a substantial 23.1% failure rate on their goal tasks. We also find that multi-turn conversations are reported as failing at 2.5 times the rate of single-turn exchanges, a pattern that reframes extended interaction as a signal of difficulty rather than engagement. We conclude by discussing the value of incorporating direct user feedback with observational data for robust LLM evaluations, and the possibilities for infrastructure that enables this goal.

Problem

• LLM evaluation
• user-defined outcomes
• interaction trajectories
• task intent
• evaluation infrastructure

Innovation

• community-centered evaluation
• conversation trajectory
• user-defined outcomes
• LLM evaluation infrastructure
• task-intent alignment