Naturalness-guided Manifold Flow Matching for Sign Language Production

2026-09-27

AI Summary

This study addresses the distortion issues in sign language generation caused by linear interpolation deviating from joint rotation manifolds and neglecting motion distributions. To this end, we propose SignNMFlow, a novel framework that reformulates sign language generation from a manifold transport perspective for the first time. Specifically, it constructs conditional paths on the motion manifold, integrating closed-form geodesic computation with learnable deviations. By minimizing kinetic energy and incorporating a naturalness metric, the method effectively couples geometric efficiency with data distribution alignment. Experimental results demonstrate that the proposed framework significantly enhances both the fidelity and naturalness of generated sign language, with qualitative and quantitative evaluations consistently validating its effectiveness.

Abstract

Sign Language Production (SLP) aims to generate sign motions from text. Conditional Flow Matching methods have achieved strong performance in SLP by constructing conditional paths that transform a source distribution into a target distribution. However, existing methods construct these paths via linear interpolation, whereas the rotational geometry of human joints confines valid joint rotations to a manifold embedded in Euclidean space. Consequently, linear interpolation between two sign motions leaves this manifold and ignores the motion distribution on it. In this paper, we revisit SLP from the perspective of manifold transport and propose a Naturalness-guided Manifold Flow Matching framework, termed \textbf{SignNMFlow}, which constructs conditional paths directly on the motion manifold by jointly considering geometric efficiency and the motion distribution. Specifically, we exploit the intrinsic geometry of the manifold and introduce a motion naturalness measure to characterize the motion distribution. By minimizing the kinetic energy under this measure, we learn a naturalness-guided interpolation that couples a closed-form geodesic, which provides geometrically efficient transport, with a learnable deviation that incorporates the motion distribution, thereby significantly improving the fidelity of generated sign motions. Extensive qualitative and quantitative evaluations demonstrate the effectiveness of this work.

Problem

• Sign Language Production
• Manifold Flow Matching
• Linear Interpolation
• Rotational Geometry
• Motion Naturalness

Innovation

• Sign Language Production
• Manifold Flow Matching
• Motion Naturalness
• Geodesic Interpolation
• Conditional Flow Matching