S$^3$: Structured Sparsity Specification

2026-04-13

AI Summary

This work addresses the challenge of unifying diverse structured sparsity patterns for efficient model compression and acceleration. The authors propose S³, an algebraic framework that formally integrates three core components—View (tensor reshaping), Block (atomic pruning units), and Scope (sparsity decision range)—to express a wide spectrum of sparsity patterns, ranging from fine-grained N:M sparsity to coarse-grained channel pruning, within a single formalism. Notably, S³ enables cross-tensor collaborative sparsification. Building upon this framework, the authors incorporate Optimal Brain Damage and Surgeon algorithms to develop structured variants of OBS/OBD. These methods significantly outperform current state-of-the-art second-order heuristic approaches in terms of output reconstruction accuracy.

Technology Category

• Machine Learning: Structured Learning
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We introduce the Structured Sparsity Specification (S$^3$), an algebraic framework for defining, composing, and implementing structured sparse patterns. S$^3$ specifies sparsity through three components: a View that reshapes the tensor via layout composition, a Block specification that defines the atomic pruning unit, and the sparsity decision Scope. Both Block and Scope support Coupling across tensors for coordinated sparsification. S$^3$ enables precise specification of diverse sparsity structures, from fine-grained N:M patterns to coarse channel pruning, and integrates seamlessly with Optimal Brain Damage (OBD) and Surgeon (OBS). We formalize the framework mathematically, demonstrate its expressiveness on canonical patterns, and validate it experimentally via structured OBS and OBD implementations built entirely on S$^3$, which surpasses well-established second order heuristics on output reconstruction across common configurations.

Problem

• structured sparsity
• sparse patterns
• tensor pruning
• model compression
• sparsity specification

Innovation

• Structured Sparsity
• Algebraic Framework
• Tensor Coupling
• Optimal Brain Damage
• N:M Sparsity