Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design

2026-07-28

AI Summary

Current robotic manipulation performance is fundamentally constrained by embodiment design, and there remains a lack of general methods capable of jointly optimizing both structure and control for tracking desired end-effector trajectories. This work proposes a unified framework based on diffusion Transformers that employs RoboTokens to jointly represent robot morphology, state, and actions. By introducing a dynamics self-guidance mechanism, the approach transforms reward-free dynamics predictions into value predictions conditioned on specific rewards, enabling zero-shot optimization for unseen trajectories and reward functions. The method facilitates cross-embodiment shared modeling and substantially outperforms evolutionary algorithm baselines across multiple design spaces. A physically fabricated ALOHA robot optimized using this framework achieves over a 70% reduction in trajectory tracking error.

Abstract

An often overlooked factor of robot manipulation performance is the embodiment of the robot itself. Motivated by this problem, we study motion-conditioned robot co-design, where the goal is to generate complete robot designs that track target end-effector trajectories (from human demonstrations) while optimizing user-defined rewards. We introduce Transformer Transformer, a diffusion transformer trained on RoboTokens, a unified tokenization of robot embodiments, states, and actions. The same architecture can be used across embodiment spaces (e.g., wheeled bimanual, quadrupeds, humanoids) and use cases (embodiment generation, cross embodiment controller). Rather than overfitting to one reward function, Transformer Transformer is a dynamics model, whose reward-agnostic state and action predictions can be converted into reward-specific value predictions. These value predictions are used to steer embodiment diffusion towards high value robot designs, through a procedure we call Dynamics Self-Guidance. Experiments across multiple design spaces show zero-shot optimization of unseen rewards and trajectories, improving performance and runtime over the evolutionary baseline. Finally, we fabricated an optimized ALOHA design, which reduced tracking error by over 70% compared to the original design.

Problem

• robot co-design
• motion-conditioned design
• embodiment optimization
• trajectory tracking
• reward-driven design

Innovation

• diffusion transformer
• robot co-design
• RoboTokens
• Dynamics Self-Guidance
• embodiment generation