DualAnchor: Preserving Language Priors and Improving Lexical Fidelity in Gloss-Free Sign Language Translation

2026-07-29

AI Summary

Current large language model (LLM)-driven approaches to gloss-free sign language translation often suffer from degraded linguistic priors and insufficient lexical fidelity, resulting in generated text that lacks fluency and distorts fine-grained details. To address this, this work proposes the DualAnchor framework, which jointly optimizes translation fluency and visual faithfulness through a dual-anchor mechanism. On one hand, it leverages the autoregressive prior of a frozen LLM to perform token-level regularization, stabilizing language generation. On the other hand, it introduces a soft visual-textual alignment module based on entropy-regularized partial optimal transport (OTA), enabling fine-grained token matching under a cosine cost. Experiments demonstrate state-of-the-art performance on PHOENIX-2014T and CSL-Daily benchmarks, where token-level prior anchoring (TPA) substantially enhances fluency and OTA effectively reduces lexical errors.

Abstract

Recent advances in large language models (LLMs) have led sign language translation (SLT), the task of converting sign-language videos into spoken-language text, to increasingly adopt LLMs as textual backbones. However, despite their strong language modeling capabilities, existing LLM-based SLT methods often undermine rather than exploit this language prior, producing disfluent translations, a failure we term language-prior degradation. Meanwhile, existing methods typically align videos and text at the sentence level, which does not ensure accurate lexical details and creates a lexical fidelity gap. To address both issues, we propose DualAnchor, a gloss-free LLM-based SLT training framework that couples two complementary anchors for linguistically fluent and visually faithful generation. Token-level Prior Anchoring (TPA) preserves the LLM's language prior by regularizing the multimodal decoder at each decoding step toward the next-token distribution of a frozen LLM conditioned on the same autoregressive prefix. Optimal Transport Alignment (OTA) improves lexical fidelity by formulating visual-textual matching as entropy-regularized partial optimal transport, with Sinkhorn optimization inducing a soft alignment between visual tokens and textual content tokens under a cosine cost. DualAnchor achieves strong overall performance on both PHOENIX-2014T and CSL-Daily. Targeted analyses attribute these gains to the complementary effects of the two anchors: TPA improves fluency, whereas OTA reduces fine-grained lexical errors.

Problem

• language-prior degradation
• lexical fidelity gap
• gloss-free sign language translation
• multimodal alignment
• large language models

Innovation

• Token-level Prior Anchoring
• Optimal Transport Alignment
• Gloss-free Sign Language Translation
• Language Prior Preservation
• Lexical Fidelity