WHTMix: Efficient Stereo Depth Estimation via Walsh-Hadamard Token Mixing

2026-07-27

AI Summary

This work addresses the high computational cost of global self-attention in Transformers for high-resolution stereo depth estimation, which hinders low-latency deployment. The authors propose, for the first time, replacing joint self-attention with a data-independent Walsh–Hadamard transform in stereo matching, achieving global token mixing with log-linear complexity while preserving the ability to model cross-view correspondences. To further enhance accuracy for distant objects, they introduce a hybrid logarithmic disparity loss. Evaluated on synthetic driving datasets, the method attains endpoint error comparable to baseline approaches while reducing computational cost by 2.46× and decreasing per-image inference latency by 2.65×.

Abstract

Stereo depth estimation for driving, robotics and augmented reality must run at high resolution under tight latency budgets, yet in transformer-based matchers the global self-attention that aggregates scene context grows quadratically with the number of pixels and comes to dominate runtime. We show that the joint self-attention stage of a stereo transformer, whose role is to spread context across both views, can be replaced by a data-independent Walsh-Hadamard token mixer that mixes tokens globally in the transform domain at log-linear cost, while the data-dependent cross-attention that performs left-right correspondence is retained. On synthetic driving data the mixer matches the attention baseline in end-point error while reducing model compute by a factor of 2.46 and single-image inference latency by a factor of 2.65. A complexity analysis shows the benefit is governed by the ratio of sequence length to channel width, which explains why high-resolution stereo matching is a particularly favorable setting and why classification transformers are not; we confirm this token-to-channel scaling on non-stereo long-sequence benchmarks. Furthermore, we introduce a hybrid log-disparity loss function designed to up-weight small-disparity pixels corresponding to long-range objects. This approach reduces the error on distant objects without incurring any additional computational overhead.

Problem

• stereo depth estimation
• transformer
• global self-attention
• computational complexity
• high-resolution

Innovation

• Walsh-Hadamard Transform
• Stereo Depth Estimation
• Token Mixing
• Efficient Transformers
• Log-Disparity Loss