🤖 AI Summary
Traditional autoregressive language models often suffer from output instability, repetition, and sensitivity to decoding strategies due to premature token discretization at each generation step. This work proposes a continuous autoregressive generation framework that represents tokens as continuous vectors and allows them to undergo multi-step dynamic evolution before committing to a hard decoding decision. For the first time, this approach enables autoregressive text generation without per-token sampling, relying instead solely on deterministic argmax operations—augmented optionally by stochastic dynamics and historical smoothing mechanisms. Once converged, the method produces coherent and diverse text while significantly reducing reliance on explicit sampling or auxiliary stabilization techniques.
📝 Abstract
Standard autoregressive language models collapse uncertainty at every generation step by committing to discrete tokens through immediate sampling. This premature discretization underlies well-known failure modes, including degenerate repetition loops in greedy decoding and a heavy reliance on heuristic sampling strategies. We introduce \textbf{Token Maturation}, a continuous autoregressive framework in which tokens evolve as vector-valued trajectories prior to discretization. Rather than sampling from a categorical distribution at each step, the model resolves uncertainty through a deterministic dynamical process in embedding space, deferring discrete commitment until the representation has geometrically stabilized. We show that this formulation mitigates degeneration \emph{intrinsically}: Token Maturation generates coherent and diverse text under fully deterministic decoding (argmax), without repetition penalties, temperature scaling, or stochastic sampling. Moreover, we identify a novel convergence behavior in which token representations stabilize spatially while predictive entropy remains high, challenging the common assumption that commitment requires probability concentration. We propose continuous token dynamics with delayed commitment as an alternative formulation of autoregressive generation that exposes structural regularities obscured by immediate discretization.