reversible transformations

In policy distillation, student models often exhibit performance degradation—termed “context-induced degradation”—when privileged context is reintroduced after being internalized during training. To address this, this work introduces the notion of “context removability” and proposes a lightweight consistency regularization method. The approach anchors the context-free output via a stop-gradient operation and enforces alignment by penalizing deviations in the context-conditioned output through a forward KL divergence computed with a single additional forward pass. Evaluated across diverse domains and architectures, the method substantially mitigates context-induced degradation: it reduces context harm in 11 out of 12 configurations, frequently improves contextual accuracy, and effectively curbs response length inflation.

This work systematically compares how State Space Models (SSMs) and Transformers propagate contextual representations in long-sequence modeling. We propose the first unified analytical framework—integrating centered kernel alignment, stability metrics, probing experiments, and parameter randomization—to quantify inter-layer and inter-token information flow differences. Our analysis reveals that Transformers suffer from rapid representational homogenization (over-smoothing) of early tokens due to self-attention, whereas SSMs preserve representational diversity initially and converge gradually in deeper layers. Crucially, Transformer inductive bias arises primarily from architectural design, while SSM behavior is predominantly shaped by training dynamics. This study provides the first principled, interpretable characterization of fundamental representational divergence between these architectures, yielding actionable design principles and optimization guidelines for long-context modeling. (149 words)

In real-world machine learning deployment, models require continual updating, composition, and selective unmerging—yet existing approaches suffer from task interference, catastrophic forgetting, and irreversible model merging. This paper proposes an Orthogonal Incremental Model Fusion (OIMF) framework: it encodes task-specific models as incremental representations within mutually orthogonal subspaces, enabling interference-free, scalable, and fully reversible model composition via modular incremental merging and orthogonality constraints. OIMF integrates incremental encoding, orthogonal projection, gradient-optimized fusion, elastic weight consolidation, and synthetic replay to ensure stability and regulatory compliance. Evaluated on multi-task vision and NLP benchmarks, OIMF significantly improves accuracy, backward transfer, and unmerging fidelity—while maintaining memory and computational efficiency—and offers theoretical interpretability grounded in subspace orthogonality.

This work proposes a novel approach to machine unlearning that efficiently and reliably fulfills privacy compliance and robustness requirements by learning a representation transformation in the model’s latent space, rather than directly modifying model parameters. Leveraging the information bottleneck principle, the method compresses information pertaining to data to be forgotten while preserving that of retained data. To the best of our knowledge, this is the first application of the information bottleneck framework to machine unlearning, enabling both zero-shot and supervised unlearning without retraining. A tractable objective function is derived via variational approximation of mutual information, allowing end-to-end optimization. Extensive experiments demonstrate that the proposed method significantly outperforms parameter-modification-based approaches across multiple benchmarks, achieving superior performance in unlearning reliability, utility preservation, and computational efficiency.

The “black-box” nature of large language models hinders rigorous performance verification. Method: We propose the first framework unifying mechanistic interpretability with formal performance verification: via weight-level mechanistic reverse-engineering, we decompose small-scale Transformer behavior on Max-of-K tasks into human-understandable algorithms and generate compact, machine-verifiable mathematical proofs (e.g., accuracy lower bounds). Contribution/Results: This establishes the first end-to-end closed loop from mechanistic understanding to formal proof. We discover that proof length positively correlates with both mechanistic insight depth and bound tightness, and identify “structural deficiency errors”—gaps between inferred mechanisms and true computational structure—as the key bottleneck limiting proof conciseness and fidelity. Validated across 151 random seeds and 4 values of K, our framework constructs 102 distinct strategies; empirical results confirm that shorter proofs reflect deeper mechanistic understanding, while higher-fidelity interpretations yield tighter performance bounds.

This work addresses the challenge of efficiently and safely removing specific concepts from generative models without affecting unrelated content. It proposes a training-free, closed-form linear transformation framework that achieves concept erasure through a two-step analytical projection: first computing a proxy projection of the target concept, then applying a constrained transformation within its left null space. As the first deterministic, geometrically interpretable, and non-iterative method for concept editing, it accomplishes erasure in just seconds on Stable Diffusion variants and FLUX models. The approach matches or exceeds state-of-the-art performance while significantly improving computational efficiency and better preserving the integrity of non-target concepts.

This work addresses the significant performance degradation of spatiotemporal forecasting models on graph-structured data under spatiotemporal distribution shifts. To mitigate this issue, the authors propose the Reversible Residual Normalization (RRN) framework, which uniquely integrates invertible transformations with graph-aware normalization. RRN constructs a bidirectional invertible architecture by combining spectral-constrained graph neural networks, centering normalization, and reversible residual modules, enabling spatially aware adaptive correction while preserving invertibility. The framework is model-agnostic and allows recovery of original distributional characteristics from the normalized latent space. Extensive experiments demonstrate that RRN substantially enhances generalization and robustness across multiple spatiotemporal prediction tasks.

This work addresses the challenge of efficiently training hundred-billion-parameter sparse Mixture-of-Experts (MoE) large language models on a single eight-GPU node, substantially reducing memory and optimizer state overhead. The authors propose a unified training framework that, for the first time, integrates reversible recurrent architectures, state-preserving progressive expansion, quantized base expert weights with low-rank adapters (TQP), Top-12 routing, and an 8K-token context window. Starting from a dense small model, this approach enables end-to-end evolution into a 120B-parameter MoE model with 5.93B activated parameters. The resulting model achieves a training loss of 1.78 and demonstrates strong multilingual (including Indic languages) and code capabilities. The model, tokenizer, and training code are publicly released.

This work addresses the problem of provenance shift—performance degradation under out-of-distribution scenarios caused by changes in the relationship between data sources and labels during deployment. It formally establishes, for the first time, the theoretical connection between provenance shift and counterfactual invariance within the framework of invariant learning, and proposes a learning objective tailored for robustness. The core contributions include the development of DeconDTN-Toolkit, the first open-source toolkit enabling simulation and mitigation of provenance shift; the introduction of a novel evaluation metric for out-of-distribution robustness; and systematic experiments that expose the fragility of empirical risk minimization approaches while demonstrating the effectiveness of the proposed strategy in enhancing model robustness.