🤖 AI Summary
This study addresses a critical issue in large language model (LLM) agents wherein tentative statements are erroneously solidified into definitive facts during memory compression, leading subsequent reasoning to rely uncritically on unverified information. The authors identify this “confidence fabrication” phenomenon as stemming from the linguistic certainty of phrasing rather than the reliability of the source itself. Through multi-turn dialogues, memory mechanism analysis, and adversarial memory injection, they demonstrate that a single misleading memory can induce erroneous judgments, yet introducing just one additional corroborating source suffices to restore accurate decision-making. Furthermore, conventional approaches such as trust labels or explicit negation instructions prove not only ineffective but potentially detrimental, exacerbating the problem rather than mitigating it.
📝 Abstract
LLM agents carry conclusions across steps and sessions in compressed memory, and memory products (e.g., mem0, LangMem) rewrite conversation into stored "facts" that later steps trust. We show this rewriting manufactures confidence: across our constructed agent settings, a casual, hedged remark becomes a confident, dated assertion the agent then obeys like a verified fact, granting every above-clearance request it faces. No attacker is needed: a role that was true once and never corrected is stored as a flat fact and acted on like a deliberate injection. We then isolate what the agent responds to. It is not the source: attributed, unattributed, and even forged "system of record" claims all grant alike. It is the confidence of the phrasing. A hedge is discounted, a flat assertion is obeyed, and this holds with no special keyword. Not all hedges are equal, though: the evidential register is the least-discounted, with "reportedly" obeyed like a flat assertion on most models. The obvious fixes fail. A passive "unverified" tag is ignored, and an active "do not trust this" instruction escalates even correct memory, so it is safe only by refusing to decide. The real fix lives in the store: keep the tentative phrasing rather than upgrade it. But that is hygiene, not a defense against an attacker who can simply write a confident lie. The deployable lesson is narrower and constructive: a single load-bearing memory is the hazard, and one redundant source restores correct decisions. We release the harness and demonstrations.