Eddy-VL 1.9B: Structural Pruning and Layered Distillation for Edge-Deployable Multimodal Embedding

2026-07-15

AI Summary

This work addresses the challenge of deploying large vision-language models on resource-constrained edge devices by introducing Eddy-VL, a lightweight 1.9B-parameter multimodal embedding model enabling offline, low-latency visual-linguistic retrieval. The authors compress the Qwen3-VL-Embedding-2B teacher model through a structured pruning strategy guided by linear centered kernel alignment (CKA) probes between adjacent layers, combined with hierarchical knowledge distillation featuring overlapping teacher-student mapping and multi-dimensional Matryoshka loss. The method further integrates FlashAttention-2 for acceleration and a joint MSE–cosine similarity loss. This approach reduces model parameters by 9.5% and inference latency by approximately 10%, while preserving 91.7% of the teacher’s performance on MMEB-V2 (63.2 vs. 68.9), with competitive results across multiple composite tasks.

Abstract

In this report, we introduce Eddy-VL 1.9B, a compressed multimodal embedding model built on Qwen3-VL-Embedding-2B for offline, edge-deployable vision-language retrieval. Eddy-VL targets air-gapped forensic and investigative settings where cloud APIs are unavailable and low latency is essential. Compression combines (i) probe-driven structural pruning that removes four redundant text-decoder layers (28 to 24) ranked by adjacent-layer linear CKA, and (ii) layered knowledge distillation with hole-covering teacher-student mappings, mid-layer attention-map 1-CKA, and final-layer MSE and cosine losses with Matryoshka dimensions {128, 256, 512, 1024, 2048}. The released model contains 1,926,188,032 parameters (3.85 GB bf16), representing approximately 9.5% fewer parameters than the 2.13B teacher model. Empirical evaluations on MMEB-V2 (78 tasks, VLM2Vec protocol) show that Eddy-VL achieves an overall score of 63.2 compared with 68.9 for the teacher, retaining 91.7% of the teacher's performance while recovering 6.4 of the 12.1 points lost through pruning alone (56.8). Compositional reasoning performance remains close to the teacher on SugarCrepe (86.1 vs. 86.4), MR2-Bench (24.5 vs. 24.7), and ARO (59.5 vs. 60.4), while Winoground group performance (6.8 vs. 8.5) remains the primary limitation. Depth pruning also reduces forward latency by approximately 10% (150.0 to 136.4 ms per image on NVIDIA DGX Spark using FlashAttention-2). We present the architecture, compression methodology, training procedures, and evaluation results, demonstrating the effectiveness of Eddy-VL for multimodal retrieval under constrained edge deployment. Model weights and inference code are publicly available on Hugging Face.

Problem

• edge deployment
• multimodal embedding
• vision-language retrieval
• model compression
• low latency

Innovation

• structural pruning
• layered knowledge distillation
• multimodal embedding
• edge deployment
• Matryoshka representation