HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories

2024-12-22

arXiv.org

AI Summary

This work addresses the bottleneck in hypernetwork training that relies on per-sample ground-truth weights. We propose HyperNet Field, a novel paradigm that models task-network weights as an input-conditioned continuous neural field, implicitly learning their optimization trajectory—not just the final converged state. Our method enables end-to-end training solely via gradient consistency constraints, eliminating the need for any sample-level weight supervision. Key technical components include neural field parameterization, implicit trajectory modeling, and gradient matching. The framework unifies support for personalized image generation and single-image or single-point-cloud-driven 3D reconstruction. Experiments demonstrate competitive performance across diverse tasks, establishing the first hypernetwork training approach that operates entirely without sample-level weight supervision.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

To efficiently adapt large models or to train generative models of neural representations, Hypernetworks have drawn interest. While hypernetworks work well, training them is cumbersome, and often requires ground truth optimized weights for each sample. However, obtaining each of these weights is a training problem of its own-one needs to train, e.g., adaptation weights or even an entire neural field for hypernetworks to regress to. In this work, we propose a method to train hypernetworks, without the need for any per-sample ground truth. Our key idea is to learn a Hypernetwork `Field` and estimate the entire trajectory of network weight training instead of simply its converged state. In other words, we introduce an additional input to the Hypernetwork, the convergence state, which then makes it act as a neural field that models the entire convergence pathway of a task network. A critical benefit in doing so is that the gradient of the estimated weights at any convergence state must then match the gradients of the original task -- this constraint alone is sufficient to train the Hypernetwork Field. We demonstrate the effectiveness of our method through the task of personalized image generation and 3D shape reconstruction from images and point clouds, demonstrating competitive results without any per-sample ground truth.

Problem

• Training hypernetworks without per-sample ground truth weights
• Learning weight trajectories instead of converged states
• Enabling efficient large model adaptation and generative modeling

Innovation

• Trains hypernetworks without per-sample ground truth
• Learns weight trajectories instead of converged states
• Uses gradient matching constraint for training