Where the Evidence Lives: Auditing AI Companions' Self-Descriptions
This study addresses the challenge that self-descriptions of AI companions and user experience ratings fail to verify the authenticity of their underlying mechanisms. We propose the first actionable auditing framework that cross-references agent self-reports, user judgments, and system implementation logs. By integrating behavioral analysis, user feedback, and log inspection for multi-source data triangulation, this framework systematically evaluates the evidential support for each claimed capability. Empirical findings reveal that certain unexecuted memory layers still receive high user ratings, exposing a significant disconnect between fluent self-descriptions and practically ineffective mechanisms. This work establishes a methodological foundation for transparently evaluating the capability boundaries of AI systems, underscoring the critical role of evidence visibility in validating their actual functionality.