LS-AR: Future-Predictive Latent Steering in Autoregressive LLMs

2026-10-02

AI Summary

This study addresses the confusion between macroscopic objectives and contextual noise in autoregressive models caused by their single-channel architecture. To this end, we propose a dual-channel decoupled framework that introduces a static goal encoder and a dynamic state tracker. By leveraging Feature-wise Linear Modulation (FiLM) conditioning, the method effectively separates continuous goal guidance from discrete token decoding. Furthermore, latent-space dominant control is incorporated to defend against text prompt injection attacks. Experimental results demonstrate that the proposed architecture achieves 100% long-horizon retrieval recall and an 89% planning completion rate, while increasing throughput by 35% and reducing GPU memory consumption by 52.8%. These findings indicate that our approach significantly enhances the robustness of long-horizon planning in autoregressive models.

Abstract

Standard autoregressive (AR) models process high-level task instructions, state history, and transient tokens within a single shared sequence of tokens. Consequently, they lack the architectural mechanisms needed to isolate macro-objectives from context noise. To overcome this single-channel limitation, we introduce Latent-Steered Autoregressive (LS-AR), a dual-channel architecture that decouples continuous goal steering from discrete token decoding via FiLM conditioning. We evaluate a Static Goal Encoder (P_0) for persistent macro-objective retention across long rollouts and a Dynamic State Tracker (P_t) for recurrent latent updates during generation. On long-horizon retrieval past context limits (H=1024, W=500), LS-AR (Static) achieves 100% target recall where parameter-matched baselines collapse (0%), while increasing throughput by ~35% and cutting peak VRAM by 52.8%. In Blocksworld planning under forced perturbations (k=1), LS-AR (Dynamic) sustains an 89.0% completion rate vs. 71.0% for the baseline, though zero-shot entity scaling (N -> N+1) exposes single-vector capacity limits (0%). Finally, dual-channel authority analysis shows that text goal dropout establishes latent-dominant control, offering structural defence against text prompt injection while introducing a latent vector attack surface.

Problem

• autoregressive models
• macro-objective isolation
• context noise
• long-horizon tasks
• single-channel limitation

Innovation

• Dual-channel architecture
• FiLM conditioning
• Latent steering
• Autoregressive LLMs
• Prompt injection defense