LAET: A Layer-wise Adaptive Ensemble Tuning Framework for Pretrained Language Models

2025-11-14

AI Summary

Financial large language models (LLMs) suffer from high computational overhead and deployment challenges. Method: This paper proposes LAET (Layer-wise Adaptive Ensemble Tuning), a parameter-efficient fine-tuning framework that introduces the first hidden-state contribution-based layer importance assessment mechanism. LAET dynamically identifies critical layers for selective fine-tuning while freezing low-contribution layers—requiring no architectural modification or knowledge distillation, and relying only on lightweight analysis. Contribution/Results: On multiple financial NLP benchmarks, the 3B-parameter LAET-finetuned model substantially outperforms strong baselines—including BloombergGPT and FinMA—and matches or exceeds GPT-4’s performance, while reducing inference cost by over an order of magnitude. LAET establishes a new paradigm for deploying high-performance financial AI under resource constraints.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Natural Language Processing (NLP) has transformed the financial industry, enabling advancements in areas such as textual analysis, risk management, and forecasting. Large language models (LLMs) like BloombergGPT and FinMA have set new benchmarks across various financial NLP tasks, including sentiment analysis, stock movement prediction, and credit risk assessment. Furthermore, FinMA-ES, a bilingual financial LLM, has also demonstrated strong performance using the FLARE and FLARE-ES benchmarks. However, the high computational demands of these models limit the accessibility of many organizations. To address this, we propose Layer-wise Adaptive Ensemble Tuning (LAET), a novel strategy that selectively fine-tunes the most effective layers of pre-trained LLMs by analyzing hidden state representations while freezing less critical layers. LAET significantly reduces computational overhead while enhancing task-specific performance. Our approach shows strong results in financial NLP tasks, outperforming existing benchmarks and state-of-the-art LLMs such as GPT-4, even with smaller LLMs ($sim$3B parameters). This work bridges cutting-edge financial NLP research and real-world deployment with efficient and scalable models for financial applications.

Problem

• Reducing computational demands of large financial language models
• Optimizing layer selection for efficient fine-tuning in NLP
• Enhancing task performance while minimizing resource requirements

Innovation

• Selectively fine-tunes most effective layers of LLMs
• Freezes less critical layers to reduce computation
• Enhances performance with smaller parameter models