Problem
Research questions and friction points this paper is trying to address.
Hypernetwork
On-the-fly Weight Generation
Few-shot Learning
ARC-1D
Specialist Models
On-the-fly Weight Generation: A Hypernetwork Proof of Concept on ARC-1D
🤖 AI Summary
This study addresses the limitation of few-shot learning approaches that typically rely on task-specific training, making it difficult to directly acquire specialized model parameters from limited demonstrations. To overcome this, we propose a hypernetwork-based context weight generation mechanism that synthesizes micro-expert model weights on the fly without requiring explicit task identifiers, thereby enabling rapid compilation and execution for few-shot tasks. Experimental evaluations on the ARC-1D benchmark validate the feasibility of dynamic weight generation and demonstrate that structured weight spaces effectively support compositional generalization. Consequently, the proposed approach achieves functional generalization capabilities that extend beyond the training distribution, offering a promising paradigm for adaptive few-shot learning.
📝 Abstract
General-purpose models can adapt to many tasks from context, while specialised models can execute individual functions with less capacity. Yet obtaining such specialists requires task-specific training or adaptation. We ask whether they can instead be generated directly from a few demonstrations. Using ARC-1D as a controlled testbed, we show that individual transformations can be represented by tiny specialist models, and that a hypernetwork can generate their parameters from context. The generated parameters form a structured weight space, while the resulting specialists show partial compositional generalisation and generalisation to transformations not seen during training. In both settings, removing explicit task identifiers improves generalisation beyond the training transformations. Together, these results provide a proof of concept that few-shot task context can be compiled on-the-fly into compact executable model parameters, and that the resulting weight space can support reuse and generalisation beyond known functions.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Hypernetwork
On-the-fly Weight Generation
Few-shot Learning
Compositional Generalization
ARC-1D
🔎 Similar Papers
No similar papers found.
