🤖 AI Summary
This study addresses the limitation that classification trees generated by generative AI, despite appearing plausible, often suffer from leaf-node redundancy and cross-branch leakage, thereby lacking practical operational utility. To overcome the constraints of conventional local naming checks, this work proposes a holistic tree-level evaluation framework incorporating a structural discriminator and a team partitionability metric. Furthermore, an agent-based framework is designed to process proprietary customer feedback corpora, combining statistical analysis with semantic deduplication for multidimensional quantitative assessment. Experimental results reveal that 97.7% of generated leaf nodes duplicate ancestor names, accompanied by severe cross-branch leakage, demonstrating that surface plausibility alone cannot guarantee taxonomy quality. These findings establish a new paradigm for evaluating the practical utility of classification systems.
📝 Abstract
Taxonomies are the symbolic representations through which AI systems organize evidence, aggregate patterns, and answer questions over large document collections. Over customer feedback, the category tree decides how every record is counted and routed, which problems get seen, and which team owns them. Agentic harnesses now make it easy to generate a plausible-looking hierarchy, and such trees are checked today with generic, individually scoped checks: each name fits its description, sits under the right parent, and stays distinct from its siblings. We ask a more operational question: when is a generated hierarchy actually useful as a production taxonomy? We build six taxonomies over two proprietary feedback corpora (1,940 and 5,000 records): for each corpus, a production reference and two repeated runs of the same harness under identical inputs. All six pass every generic naming and structure check, and a deeper product-coverage check even prefers the generated trees. Yet in every generated tree at least 97.7% of leaf names merely restate an ancestor's name (13.9% and 2.9% in the references), and in one, three of every four records fall under multiple top-level categories. We introduce two families of whole-tree metrics: structural discriminators test whether a tree's shape was learned from the data or imposed by its generator; team partitionability tests whether branches split feedback into groups teams can own. Trees the generic checks rate as equally correct differ by 27 percentage points in cross-branch leakage, and only one of the two beats a random split. Surface plausibility is an insufficient measure of taxonomy quality: evaluation must measure the whole tree as well as each node.