Human Grounded Evaluation of Large Language Models for Optical Network Automation

2026-07-20

AI Summary

This work addresses the significant disparity in output quality and inference cost among large language models (LLMs) in optical network automation, as well as the lack of efficient, scalable evaluation methodologies. To tackle these challenges, the authors propose HuGLEN, an innovative evaluation framework that synergistically combines limited expert annotations with an LLM-as-a-judge mechanism. HuGLEN employs a reproducible, tiered evaluation pipeline augmented with explainable AI (XAI)-based output interpretation and a quantitative Quality-Efficiency Score (QES) metric, enabling consistent model selection tailored to network operators’ needs. Applied to the task of explaining optical transmission quality, HuGLEN identifies a 12B-parameter mid-sized model as achieving the optimal QES, thereby demonstrating the framework’s effectiveness and practicality in LLM evaluation and selection for real-world optical networking applications.

Abstract

Large language models (LLMs) are increasingly adopted for network automation, yet their output quality and inference cost can vary substantially across LLM families. We present HuGLEN, a stepwise evaluation pipeline that uses an LLM-as-a-judge together with a small set of expert ratings to enable scalable and reproducible comparison of candidate LLMs, and to rank them using a quality efficiency score (QES). We demonstrate HuGLEN for translating outputs from an explainable artificial intelligence (XAI) model for the optical network quality of transmission (QoT) estimation task into operator-friendly explanations. Our results show that a medium-sized LLM (12B parameters) achieves the highest QES, indicating the best trade-off between explanation quality and efficiency. Overall, HuGLEN reduces the human-labeling burden while supporting consistent model selection for operator-facing automation tasks.

Problem

• Large Language Models
• Optical Network Automation
• Model Evaluation
• Quality of Transmission
• Human Grounded Evaluation

Innovation

• LLM-as-a-judge
• quality efficiency score
• human grounded evaluation
• optical network automation
• explainable AI