Beyond Real Weights: Hypercomplex Representations for Stable Quantization

2025-12-09

AI Summary

To address the excessive parameter count and low deployment efficiency of multimodal large language models (MLLMs) caused by high-dimensional vision-language alignment, this paper proposes a progressive hypercomplex reparameterization method: dense feed-forward layers are gradually replaced with parameterized hypercomplex multiplication (PHM) layers; residual interpolation scheduling and a lightweight reconstruction loss are introduced during training, jointly enabling structural compression via knowledge distillation. The approach preserves multimodal alignment capability and behavioral fidelity to the original model while natively supporting quantization. Experiments across multiple mainstream vision-language models demonstrate that the compressed models achieve significant reductions in parameter count and inference latency, matching the original performance while accelerating inference by up to 2.1×. The core contribution lies in the first systematic application of progressive hypercomplex substitution to MLLM efficiency optimization, achieving a balanced trade-off among accuracy, computational efficiency, and deployment friendliness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Multimodal language models (MLLMs) require large parameter capacity to align high-dimensional visual features with linguistic representations, making them computationally heavy and difficult to deploy efficiently. We introduce a progressive reparameterization strategy that compresses these models by gradually replacing dense feed-forward network blocks with compact Parameterized Hypercomplex Multiplication (PHM) layers. A residual interpolation schedule, together with lightweight reconstruction and knowledge distillation losses, ensures that the PHM modules inherit the functional behavior of their dense counterparts during training. This transition yields substantial parameter and FLOP reductions while preserving strong multimodal alignment, enabling faster inference without degrading output quality. We evaluate the approach on multiple vision-language models (VLMs). Our method maintains performance comparable to the base models while delivering significant reductions in model size and inference latency. Progressive PHM substitution thus offers an architecture-compatible path toward more efficient multimodal reasoning and complements existing low-bit quantization techniques.

Problem

• Compress multimodal language models to reduce computational load
• Replace dense network blocks with hypercomplex layers for efficiency
• Maintain performance while decreasing model size and inference latency

Innovation

• Progressive reparameterization with hypercomplex multiplication layers
• Residual interpolation and lightweight losses for functional inheritance
• Architecture-compatible compression for efficient multimodal reasoning