🤖 AI Summary
This work addresses a critical limitation in current personalization approaches for large language models (LLMs), which often overlook the heterogeneity of users’ domain-specific expertise, thereby risking overreliance on AI in areas where users are less competent and potentially inducing professional drift. To mitigate this, the authors propose a scaffolding intervention framework grounded in user competency profiling, dynamically modulating LLM outputs by categorizing domains into strong, mixed, and weak proficiency zones. The framework introduces, for the first time, a structured competency-profile-driven intervention mechanism that integrates typological competency partitioning, competency-conditioned response strategies, and multi-LLM ensembling. Evaluated on MMLU subsets, the approach demonstrates that swapping user profiles can invert performance categories, while selective activation in mixed zones significantly enhances human-AI collaboration reliability.
📝 Abstract
Large language model personalization typically adapts outputs to user preferences and style but does not account for differences in user evaluation capacity across domains of expertise. This limitation can encourage Professional Domain Drift, where users rely on AI generated reasoning in domains they cannot reliably evaluate. We introduce Capability Conditioned Scaffolding, a typed framework that partitions expertise into strong, mixed, and weak domains and conditions intervention behavior on structured capability profiles. A pilot evaluation across multiple MMLU subsets and four LLM substrates shows consistent profile conditioned intervention behavior, including categorical inversion under profile swapping and selective activation in mixed domain risk zones. These findings suggest that capability aware scaffolding can support more reliable professional human AI collaboration beyond stylistic personalization.