Representation Without Reward: A JEPA Audit for LLM Fine-Tuning

2026-05-14

AI Summary

This study investigates whether Joint Embedding Predictive Architectures (JEPA) can enhance downstream performance of large language models by optimizing hidden representations, with a focus on the coupling between representation learning and task-specific metrics. Using a fixed Llama-3.2-1B-Instruct model under a LoRA fine-tuning framework, the authors systematically evaluate 22 auxiliary objectives on a natural language-to-regular-expression generation task. They reformulate JEPA as a “decoder-visible” representation–performance coupling problem and conduct ablation studies incorporating trajectory shape regularization, distributional constraints, and asymmetric prediction designs. Results show that while multiple auxiliary objectives substantially alter the geometry of hidden states, none yield statistically significant improvements in task accuracy after multiple-testing correction. Although the proposed “decoder-visible” JEPA achieves alignment with cross-entropy gradients for the first time, its performance gains remain insignificant, a finding robust across both LoRA and full fine-tuning settings.

Abstract

Joint-embedding predictive architectures (JEPAs) propose that a model should learn more useful abstractions when trained to predict latent representations rather than observed outputs. For autoregressive language-model fine-tuning the principle entails a stricter requirement: the induced hidden-state geometry must reach the language-model head \emph{and} improve the decoded task metric. We test that requirement under a fixed Llama-3.2-1B-Instruct LoRA harness on natural-language-to-regex generation, comparing twenty-two training-time auxiliaries across trajectory-shape regularisation, distributional constraints, predictor/target asymmetry, Fisher-metric Jacobi residuals, and a decoder-visible JEPA objective constructed to lie in cross-entropy's positive cone. The empirical answer is a structured null: several auxiliaries clear single-cell paired $α= 0.10$ without correction (T3-Local at $Δ= +2.53$~pp, $p = 0.003$ being the strongest), but none survives Bonferroni or Holm--Bonferroni at the relevant family-wise threshold, even though many change curvature, anisotropy, variance, and gradient direction. Decoder-visible JEPA yields the first positive auxiliary--cross-entropy gradient cosine in the study, yet exact match remains inside seed noise; a full-fine-tuning replication of the same auxiliary at $n = 5$ seeds reproduces the null on both benchmarks (TURK: $Δ= +0.04$~pp, $p_{\text{paired}} = 0.96$; SYNTH: $Δ= +0.52$~pp, $p_{\text{paired}} = 0.28$), so the null is robust across LoRA and full fine-tuning for the decoder-visible construction. Hidden-state representation work and decoded-task accuracy are therefore weakly coupled in this regime; we accordingly reframe LLM-domain JEPA evaluation as a coupling problem, in which the operative question is under which metrics useful hidden geometry becomes decoder-visible task signal.

Problem

• JEPA
• LLM fine-tuning
• representation learning
• decoder visibility
• task accuracy

Innovation

• JEPA
• decoder-visible representation
• latent prediction
• representation-task coupling
• LoRA fine-tuning