FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory

2026-08-05

AI Summary

This work addresses the limitations of existing latent-variable memory approaches for GUI agents, which often lose fine-grained details during trajectory compression and struggle with multi-stage decision-making due to monolithic memory structures vulnerable to irrelevant information. To overcome these issues, we propose a novel latent memory interface trained end-to-end under a frozen policy, which uniquely decouples content storage, state-conditional retrieval, and a trust gating mechanism. Our method introduces role-aware content bases to distinguish reusable experiences from task progress, integrates both semantic and functional supervision signals, and employs a lightweight trust gate to dynamically filter relevant memories. Evaluated across five GUI agent benchmarks, our approach significantly outperforms current methods, with ablation studies confirming that each component effectively preserves complementary information, enhances contextual robustness, and suppresses interference.

Abstract

GUI agents must remember both useful experience from earlier tasks and unfinished progress in the current interaction. Latent memory offers a compact solution by compressing multimodal trajectories into a few continuous tokens. Existing methods, however, usually map each trajectory to one fixed memory block and train it mainly through next-action supervision. This creates three practical problems: important details may be lost during compression, the same memory block must serve different decision stages, and irrelevant retrieved trajectories may still mislead the agent. We introduce FocusMem, which separates these responsibilities within a compact latent-memory interface. A role-aware content basis encourages episodic memory to retain reusable experience and working memory to retain task progress. A state-conditioned readout generates a decision-specific view of the same stored evidence, while a lightweight trust gate can suppress memory blocks that appear irrelevant to the current step. All components are trained while the GUI policy remains frozen. Across five GUI-agent benchmarks, FocusMem consistently outperforms a fully matched action-only fixed-memory baseline and prior latent memory adaptations. Further analysis shows that semantic and functional supervision preserve complementary information, state-conditioned readout is more robust as surrounding trajectory context grows, and the trust gate reduces the harm caused by injected irrelevant episodic evidence. These results show that effective latent memory depends not only on compressing past interaction, but also on what is retained, what is exposed, and what is allowed.

Problem

• latent memory
• GUI agents
• memory compression
• irrelevant retrieval
• decision stages

Innovation

• latent memory
• role-aware content basis
• state-conditioned readout
• trust gate
• GUI agents