RT-2-X 55B

Google DeepMind · Oct 2023 · Research use (weights not public)

Co-fine-tunes a web-scale vision-language model on robot trajectories, emitting actions as discretized text tokens. RT-2-X adds the cross-embodiment Open X-Embodiment mixture for emergent generalization.

55B Paper Code

Success rate: 63%
Downloads: 41K
Control rate: 3 Hz
Training episodes: 1.1M

Benchmarks

Numbers as reported by the authors on the named evaluation suite.

Evaluation	Suite	Score
Emergent skill evaluation Symbol understanding, reasoning, human recognition	RT-2 eval	62%success
Unseen objects	RT-2 seen-vs-unseen	76%success
Cross-embodiment transfer 3x on emergent tasks vs RT-2	Open X-Embodiment	50%relative gain vs single-robot

Evaluation

Suite

Score

Emergent skill evaluation

Symbol understanding, reasoning, human recognition

RT-2 eval

62%success

Unseen objects

RT-2 seen-vs-unseen

76%success

Cross-embodiment transfer

3x on emergent tasks vs RT-2

Open X-Embodiment

50%relative gain vs single-robot

Training

Architecture

Vision-language-action transformer (PaLI-X backbone)

Parameters

55B

Hardware

Google TPU v4 pods

Compute

92K accelerator-hours

Training data hours

21K h

Training episodes

1.1M

Recipe

Co-fine-tuning: web VQA batches interleaved with robot action batches so language grounding is not forgotten.

Fine-tuning notes

Emergent skills degrade when fine-tuned on a single embodiment; keep 30-50% web data in the mixture.

Linked datasets, robots & papers

Resolving related models, datasets and papers…