lightweight head design

Existing multi-foundation-model feature fusion methods rely heavily on downstream fine-tuning or labor-intensive hyperparameter optimization. Method: We propose ComBo—a probe-style adapter that freezes backbone parameters and requires no backpropagation. ComBo employs a lightweight Transformer to integrate token-level compressed features from multiple models and hierarchical levels, introducing a novel multi-backbone joint probing mechanism for cross-model feature fusion and task-relevance self-adaptation. It is dataset-agnostic—requiring no task-specific hyperparameters—and preserves all foundation model parameters unchanged. Contribution/Results: On all 19 VTAB-1k tasks, ComBo significantly outperforms existing probing baselines, matches or surpasses costly distillation-based model merging approaches, and remains compatible with efficient downstream probing of fine-tuned models. ComBo establishes a new paradigm for general-purpose, efficient, plug-and-play multi-model feature composition.

This work addresses the lack of a unified and efficient backbone architecture for facial analysis by proposing a joint training framework that integrates Masked Autoencoders (MAE) as a self-supervised auxiliary task within the L-SSAT paradigm. The study systematically evaluates the compatibility of various backbone networks—ranging from shallow to deep—with fused local texture descriptors. Experimental results demonstrate that no single backbone universally outperforms others across tasks; instead, optimal backbone selection is inherently task-dependent. The approach achieves average accuracies of 0.94, 0.87, and 0.88 on FaceForensics++, CelebA, and AffectNet, respectively, underscoring the critical role of task-specific backbone design in enhancing model robustness and discriminative power.

To address the challenges of deploying large Transformer models on resource-constrained devices—namely, high computational overhead and redundancy of task-agnostic knowledge—this paper proposes a **one-shot, task-oriented parameter projection method** that directly maps pretrained model parameters to compact, task-specific models without fine-tuning or additional training data. The core innovation lies in a learnable, conditional parameter projection network that extracts semantically relevant knowledge subsets and achieves structured parameter compression guided by task semantics. Evaluated on image modeling tasks, the resulting lightweight models outperform general-purpose conditional baselines in accuracy while accelerating inference by 3.2× and reducing parameter count by 91%. This approach effectively balances predictive performance, computational efficiency, and practical deployability.

Inferring the functionality of publicly available but undocumented model weights is highly challenging due to pervasive nuisance variations—semantically irrelevant perturbations—in the weight space. To address this, we propose ProbeX, the first theory-driven probing framework specifically designed for single-hidden-layer weights and grounded in the structural properties of Model Trees. Leveraging the low-nuisance characteristic inherent to model families organized as trees, ProbeX integrates a linear classifier with a hierarchical expert mechanism to achieve efficient and interpretable mapping from weight space to language embedding space. Experiments demonstrate that ProbeX significantly outperforms existing baselines on cross-model-category prediction tasks. Notably, it achieves zero-shot language-space mapping and semantic classification of Stable Diffusion weights—the first such result—thereby establishing a novel paradigm for functional reverse-engineering of black-box models.

This work addresses the challenges of deploying Transformer models on resource-constrained edge devices, where computational complexity, memory footprint, and power consumption pose significant bottlenecks. The study systematically evaluates lightweight Transformer architectures alongside optimization strategies—including compression, quantization, pruning, and knowledge distillation—and integrates sparse attention mechanisms, mixed-precision quantization (INT8/FP16), and hardware-aware neural architecture search to enable efficient deployment within frameworks such as TensorFlow Lite and CoreML. A proposed six-step deployment pipeline achieves 4–10× model compression and 3–9× latency reduction at a power budget of 2–5 W, with accuracy degradation below 2% (retaining 75–96% of original accuracy). The analysis further uncovers a consistent memory bandwidth bottleneck, revealing that models with 15–40 million parameters attain 60–75% hardware utilization on mainstream edge platforms.

Existing single-view linear probes struggle to model the higher-order interaction structures between rows and columns in model weights, limiting the effectiveness of weight space learning. To address this, this work proposes MVProbe—the first multi-view probing framework tailored for weight representations—which explicitly captures higher-order correlations by fusing first-order signals with an interaction-aware view constructed via Gram matrices. The method introduces learnable probe vectors and incorporates a scaling-law-guided normalization strategy to enable adaptive normalization and fusion of multi-branch features. Evaluated on the Model Jungle benchmark, MVProbe consistently outperforms the current state-of-the-art ProbeX across diverse architectures, including ResNet, SupViT, MAE, DINO, and Stable Diffusion LoRA.

This work addresses the challenge of efficiently and legally auditing high-risk language models at scale for harmful specialization—such as generation of child sexual abuse material (CSAM)—without producing illicit content. The authors propose a novel non-generative evaluation paradigm that detects harmful specialization by analyzing internal model states, specifically perturbations in intermediate representations induced by LoRA adapters. Leveraging Gaussian probing techniques, the method quantifies changes in internal representations through Gaussian latent ensembles. Experimental results demonstrate that this approach reliably distinguishes between benign and harmful models in CSAM-related tasks and exhibits robustness against adversarial interventions such as weight scaling.

This work addresses the challenges of building efficient and lightweight text mining models under label-scarce and resource-constrained conditions, where conventional semi-supervised learning methods often suffer from high computational overhead and susceptibility to local optima. To overcome these limitations, the authors propose NanoNet, a novel framework that uniquely integrates online knowledge distillation, mutual learning regularization, and parameter-efficient fine-tuning to collaboratively train multiple lightweight submodels. This synergistic approach substantially reduces reliance on labeled data and training costs while simultaneously enhancing both inference efficiency and model performance. NanoNet thus offers an effective solution for low-supervision, low-latency text mining scenarios.

Llama-3.1-8B-Instruct systematically misjudges numerical comparisons (e.g., “9.11 > 9.8”) in chat/qa formats. Method: We employ attention head intervention, Sparse Autoencoder (SAE) feature analysis, and pattern substitution experiments to dissect the underlying mechanisms. Contribution/Results: We identify a modular functional specialization: even-indexed attention heads are dedicated to numerical comparison, while odd-indexed heads handle other tasks. Activating only eight even-indexed heads fully corrects the error—revealing a sharp computational threshold and functional redundancy. Furthermore, format switching triggers dynamic evolution of numerical representations—from disentangled to highly entangled—with critical features exhibiting strong amplification. Remarkably, full correction is achieved using just 25% of attention heads; 60% pattern substitution constitutes the critical threshold. This work establishes, for the first time, modular functional decomposition of numerical comparison across attention heads in Transformers, substantially enhancing model interpretability and reasoning efficiency.

To address excessive computational overhead and parameter redundancy in LiDAR point cloud 3D detectors’ backbone networks, this paper proposes Dense Backbone—a lightweight, densely connected backbone specifically designed for 3D detection. It is the first to introduce dense connectivity into 3D detection backbone architecture while preserving plug-and-play compatibility—requiring no modifications to the detection head or other components. Built upon the pillar-based representation, Dense Backbone integrates lightweight convolutions with cross-layer feature reuse mechanisms to enhance representational efficiency. Evaluated on the nuScenes benchmark, DensePillarNet—a detector incorporating Dense Backbone—achieves a 29% reduction in model parameters and a 28% decrease in inference latency compared to the baseline, with only a marginal 2% drop in mAP. This yields a significantly improved accuracy–efficiency trade-off, making it particularly suitable for resource-constrained autonomous driving deployment scenarios.