Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputs

2026-10-04

AI Summary

This study addresses the high communication overhead of parameter exchange and interference in weight space during collaborative learning by proposing a distributed subconscious learning framework. Its core innovation lies in replacing model parameter updates with the output behavior of random carrier probes as the collaboration primitive, enabling knowledge sharing and model composition without task-relevant proxy data. This approach supports both one-shot base model merging and iterative federated learning. Experimental results demonstrate that the method preserves 94.56% of LLM preferences, improves GSM8K performance by 22 points, and reduces upload volume by 30–49×. Furthermore, it achieves 96.83% accuracy on MNIST while decreasing uplink bandwidth consumption by 8.9×.

Abstract

Collaborative learning typically exchanges model parameters: federated clients communicate updates, while independently adapted foundation models are combined by exchanging adapters or checkpoints. This makes communication scale with model size and requires local specializations to be reconciled in weight space, where interference is common. We ask whether knowledge can instead be shared through model behavior on task-unrelated inputs. We introduce Distributed Subliminal Learning (DSL), a collaborative learning primitive in which participants adapt a common model locally, probe it with task-unrelated inputs, and transmit only the resulting carrier outputs. A coordinator pools these outputs and distills them into a shared model. The primitive supports one-shot foundation-model composition through carrier completions and iterative federated learning through carrier logits, without transmitting model updates or requiring task-related proxy data. In LLM composition, compared with LoRA averaging, DSL achieves higher preference retention (94.56% vs. 87.76%) and a larger GSM8K gain over the base model (22.0 vs. 0.6 points), while reducing upload by 30.6-49.0$\times$. In federated classification, DSL reaches 96.83% on MNIST with 8.9$\times$ less uplink than FedAvg and provides lower-communication operating points on CIFAR-10 and Tiny ImageNet. These results establish random-carrier outputs as a practical communication primitive for knowledge sharing across distinct collaborative learning paradigms.

Problem

• collaborative learning
• communication efficiency
• model composition
• federated learning
• knowledge sharing

Innovation

• Distributed Subliminal Learning
• Random-Carrier Outputs
• Federated Learning
• Foundation Model Composition
• Communication Efficiency