Transformer Transformer: One Model to Design and Control Robots
What if you could show a robot a demonstration and have it design itself to perform that exact task? That's exactly what Transformer Transformer does. This unified model takes a manipulation demo as input and outputs a complete robot—every link, joint, motor, and inertial property—optimized for that motion. The same model then controls the generated robot.
On a cloth-flinging task with an ALOHA2 bimanual platform, the custom-designed robot reduced tracking error by 73% and max joint speed by 30% compared to the original ALOHA2 design. These results come from a diffusion transformer trained on RoboTokens, a unified tokenization of robot embodiments, states, and actions.
RoboTokens: A Universal Robot Language
RoboTokens encode any rigid articulated robot into a typed token sequence. Five embodiment token types (links, fixed joints, rotating/sliding joints, ball joints, motors) plus state and action tokens capture everything needed to describe a robot and its dynamics. The tokenizer handles robots ranging from a 0.65 kg dexterous hand to a 67.5 kg quadruped, with 6 to 35 active joints. Each robot becomes a sequence of 28–101 tokens—27–110× more compact than MJCF text.
Tokens reference each other by ID, so the same format works for a 6-DoF arm or a 35-DoF biped. The tokenizer canonicalizes redundant spatial offsets and stores data in a single frame, reducing variance the model must learn. New token types (like target end-effector poses) slot in without architectural changes.
Transformer Transformer: One Architecture, Many Roles
The model is a diffusion transformer (DiT) with a DDIM noise schedule. By masking different token types at inference, the same network acts as an unconditional robot generator, a cross-embodiment controller, or a motion-conditioned robot designer—all with the same weights.
As a cross-embodiment controller, it behavior-clones from RL experts trained one-per-design on procedural quadrupeds. The controller adjusts to continuous variations (link dimensions) and discrete variations (DoF, knee direction) without retraining. As a generator, it produces fixed-base arms, quadrupeds, humanoids, and dexterous hands from Gaussian noise.
Dynamics Self-Guidance: Zero-Shot Reward Optimization
The key insight: since the model predicts both embodiment and dynamics, you can feed those predictions into any user-defined reward function and get a predicted reward. Because the reward is differentiable with respect to embodiment tokens, gradients can steer the diffusion process toward higher reward at every denoising step. The authors call this Dynamics Self-Guidance.
This enables zero-shot optimization for unseen rewards. For example, adding a torque-penalty term shifts the generated robot to be smaller and lighter, using 30% less average torque. Adding a size penalty shifts it again—all without retraining. The gradient flows through the entire episode horizon at once, optimizing hardware over long-horizon dynamics.
Results: Better Designs, Faster
Experiments compare against Random sampling and CMA-ES (a black-box optimizer). On a quadruped dynamic toss task, Transformer Transformer generates robots that achieve higher reward with fewer simulation steps. On the UMI bimanual dish-washing dataset, it optimizes over 26 held-out trajectories simultaneously using diffusion composition, navigating both discrete (spine designs: fixed, sliding, bending) and continuous (arm dimensions, mounting offsets) spaces.
The model fabricates one design for cloth flinging on an ALOHA2 platform, reducing tracking error by 73% and max joint speed by 30%. This is a concrete demonstration that the approach works on real hardware.
What This Means for Robotics
Transformer Transformer reframes robot co-design as a three-step process: demonstrate, generate, validate. The same network plays generator, critic, and controller by training on a unified tokenization. This eliminates separate pipelines for optimization, dynamics modeling, and control.
The paper is available at https://transformer-transformer.github.io/. The code is not yet released, but the authors promise open-source release. If you're working on robot design or manipulation, this is worth watching.
