TLC-DiT: Task-Aligned Local Visual Conditioning for Robust Multitask Robot Manipulation

2026-09-28

AI Summary

This study addresses the lack of interpretability and vulnerability to perturbations in robotic policies caused by implicitly encoded local visual evidence. To this end, we propose TLC-DiT, a plug-and-play module for multi-task diffusion Transformers. Without altering the diffusion objective, this method integrates DINOv2 patch features, CLIP task embeddings, FiLM modulation, and CoordConv adapters to explicitly construct task-aligned local visual feature maps, which are then concatenated with global conditioning. This design enables interpretable decision-making and enhanced robustness. Experiments demonstrate that TLC-DiT achieves average success rates of 93.5% and 57.24% on the LIBERO and LIBERO-plus benchmarks, respectively. Furthermore, in a real-world bimanual tea-making task, the completion rate improves substantially from 44% to 89%, significantly outperforming existing baselines.

Abstract

Language-conditioned robot policies have made clear progress in multitask manipulation, but task-relevant local visual evidence usually stays hidden inside a visual backbone or attention layers. This leaves the policy difficult to inspect and fragile under visual change, two symptoms of a missing explicit, task-aligned local visual channel. We present TLC-DiT, a plug-in extension of the Multitask Diffusion Transformer (DiT) policy that adds explicit task-guided local visual feature maps without changing the diffusion objective or the action-generation process. For each camera view, frozen DINOv2 patch features are modulated by the CLIP task embedding through FiLM and refined by a lightweight CoordConv CNN adapter into smooth spatial maps, which are concatenated with the original global image, language, joint-state, and timestep conditions. On LIBERO, TLC-DiT reaches a 93.5% average success rate, compared with 86.5% for Multitask DiT and 79.25% for SmolVLA. On LIBERO-plus, the total success rate improves from 54.07% to 57.24%, with larger gains under camera, background, and sensor-noise changes. In real-world bimanual tasks, TLC-DiT raises Teabag Putting completion from 44% to 89% while maintaining comparable Match Box Opening performance. Feature-map visualizations confirm that the model attends to task-relevant regions across views and perturbations, providing a direct way to inspect the visual evidence.

Problem

• multitask robot manipulation
• local visual conditioning
• visual robustness
• policy interpretability
• language-conditioned policies

Innovation

• Task-Aligned Local Visual Conditioning
• Diffusion Transformer
• Plug-in Extension
• FiLM Modulation
• Multitask Robot Manipulation