Layer-Informed Fine-Tuning via Three-Stage Functional Segmentation of LLMs

2026-09-28

AI Summary

This study addresses the inefficiency and lack of specificity in full fine-tuning caused by the opaque internal reasoning mechanisms of large language models (LLMs). To this end, we propose LIFT, a precise sparse fine-tuning method grounded in layer-wise functional division. This work is the first to reveal a three-stage internal architecture within LLMs comprising conceptualization, reasoning, and verbalization. By leveraging cross-lingual inference and sensitivity analysis, LIFT identifies functional bottlenecks and performs selective parameter updates exclusively on core layers. Experimental results demonstrate that LIFT significantly accelerates training while substantially enhancing model performance on complex reasoning tasks.

Abstract

In recent years, the performance of large language models (LLMs) on reasoning tasks has been remarkable, even surpassing human capabilities on various benchmarks. However, there remains a lack of clear understanding in the academic community regarding how the structure and internal parameters of LLMs progressively solve complex reasoning problems. In this study, we investigate the inference process of LLMs on cross-linguistic materials and propose the hypothesis that LLM layers exhibit a structured division of labor across conceptualization, reasoning, and textualization. Based on this hypothesis, we introduce a bottleneck identification mechanism using sensitivity analysis to pinpoint the most critical functional stage for a specific task. Leveraging this insight, we propose a novel approach, Layer-Informed Fine-Tuning (LIFT), which achieves efficient and effective fine-tuning by selectively updating only these functionally critical layers. We then conduct extensive experiments to show that the LIFT method not only accelerates the training process but also significantly improves model performance.

Problem

• Large Language Models
• Reasoning
• Functional Segmentation
• Layer-wise Analysis
• Cross-linguistic

Innovation

• Layer-Informed Fine-Tuning
• Functional Segmentation
• Sensitivity Analysis
• Bottleneck Identification
• Large Language Models