D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentation

2026-08-06

AI Summary

This work addresses the inconsistency between frame representations and action prototypes in existing optimal transport–based unsupervised action segmentation methods, particularly under ambiguous transitions or short-duration, low-frequency actions. To resolve this issue, the authors propose a dual-loop optimization mechanism that jointly refines representations and prototypes: a graph-constrained module preserves local geometric structure, while action prototypes are periodically re-estimated via k-means or optimal transport (OT) barycenters. Evaluated on five benchmark datasets, the method substantially outperforms CLOT, achieving up to a 12.7% improvement in per-video F1 score and a 10.2% gain in mIoU. Notably, it establishes the first effective unsupervised segmentation baseline on the fine-grained Assembly101 dataset.

Abstract

Optimal transport (OT) has emerged as an effective framework for unsupervised action segmentation. Yet, in existing OT-based methods, the latent action prototypes that define the OT costs are not re-estimated from the refined frame geometry. Instead, they evolve solely through gradients from the pseudo-label loss. We identify this \emph{representation--prototype inconsistency} as a central bottleneck, particularly around ambiguous transitions and for short or infrequent actions. To address this issue, we build on the recently introduced CLOT, which refines frame embeddings based on estimated segment embeddings, and further re-estimates the action prototypes from the refined frame embeddings. Specifically, we introduce a graph-constrained module that regularizes the OT-refined frame and segment representations by preserving the local neighborhood geometry of the encoder output. An action-embedding refinement step then periodically re-anchors the prototypes to this stabilized representation geometry. We study two instantiations that share the same backbone, graph module, and objective: D-CLOT updates the prototypes using $k$-means, whereas D-CLOT$_{B}$ updates them as OT barycenters weighted by the refined transport plan, yielding an assignment-aware prototype update consistent with the current transport geometry. Across five established benchmarks, both variants improve segment-level quality over CLOT, with per-video gains of up to $+12.7$ F1 and $+10.2$ mIoU (YTI) and activity-level gains of up to $+8.9$ F1 (FS-Eval). We further establish the first unsupervised action-segmentation baseline on Assembly101, a procedural and substantially more fine-grained benchmark than those commonly used in prior work. Extensive ablations and sensitivity analyses demonstrate that the two refinement mechanisms are complementary and robust.

Problem

• unsupervised action segmentation
• optimal transport
• representation--prototype inconsistency
• action prototypes
• frame embeddings

Innovation

• optimal transport
• unsupervised action segmentation
• prototype refinement
• graph-constrained representation
• double closed-loop