Dynamic Agentic AI Expert Profiler System Architecture for Multidomain Intelligence Modeling

2026-04-06

AI Summary

This work proposes a modular hierarchical architecture based on LLaMA-3.1 (8B) to enhance dynamic perception of user expertise levels in human–AI interaction. By integrating text preprocessing, multidimensional scoring, aggregation, and classification components, the system dynamically categorizes users’ natural language responses in real time into four proficiency tiers: Novice, Basic, Advanced, and Expert. The approach innovatively enables turn-by-turn assessment during dialogue and incorporates a dual-stage validation mechanism combining static and dynamic evaluation. Experimental results demonstrate that the system’s classifications align with users’ self-assessments at rates of 83%–97%, significantly outperforming conventional one-time holistic assessment methods and thereby improving both the accuracy and timeliness of context-aware adaptation.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In today's artificial intelligence driven world, modern systems communicate with people from diverse backgrounds and skill levels. For human-machine interaction to be meaningful, systems must be aware of context and user expertise. This study proposes an agentic AI profiler that classifies natural language responses into four levels: Novice, Basic, Advanced, and Expert. The system uses a modular layered architecture built on LLaMA v3.1 (8B), with components for text preprocessing, scoring, aggregation, and classification. Evaluation was conducted in two phases: a static phase using pre-recorded transcripts from 82 participants, and a dynamic phase with 402 live interviews conducted by an agentic AI interviewer. In both phases, participant self-ratings were compared with profiler predictions. In the dynamic phase, expertise was assessed after each response rather than at the end of the interview. Across domains, 83% to 97% of profiler evaluations matched participant self-assessments. Remaining differences were due to self-rating bias, unclear responses, and occasional misinterpretation of nuanced expertise by the language model.

Problem

• expertise profiling
• human-AI interaction
• multidomain intelligence
• user modeling
• natural language understanding

Innovation

• Agentic AI
• Expert Profiling
• Dynamic Assessment
• Multidomain Intelligence
• LLaMA-based Architecture