Convolution for Large Language Models

2026-07-20

AI Summary

This work addresses the lack of explicit modeling of linguistic locality in Transformer-based large language models. To incorporate a local inductive bias, the authors propose inserting lightweight depthwise separable convolutions (kernel size k=3) before the query, key, and value projections in Qwen3’s attention mechanism, accompanied by residual connections but without additional normalization or activation functions. This minimal modification introduces less than 0.01% parameter overhead yet consistently improves average accuracy across seven downstream tasks at multiple pretraining scales. Analyses reveal enhanced sensitivity to local context around repeated tokens, confirming the effectiveness of the introduced locality-aware architectural bias.

Abstract

Large language models (LLMs) largely rely on Transformers, where self-attention provides global token interaction but does not explicitly encode the locality of natural language. We study whether lightweight depthwise convolutions can supply this local inductive bias without materially increasing model size. Our macro-level ablation compares convolution at 17 locations in a Qwen3 Transformer block and finds the best results when convolution is applied to the projected queries, keys, and values before attention. A subsequent micro-level study favors a residual depthwise convolution with kernel size $k=3$, without additional normalization or activation. Across Qwen3 models and several pre-training data budgets, this design improves the average accuracy on seven downstream benchmarks while adding less than $0.01\%$ parameters. A representation-level case study further suggests that the convolution makes repeated token IDs more sensitive to their immediate context. These results support depthwise convolution as a lightweight complement to self-attention for modeling short-range token interactions.

Problem

• large language models
• locality
• inductive bias
• self-attention
• short-range interactions

Innovation

• depthwise convolution
• local inductive bias
• Transformer architecture
• large language models
• token interaction