ModAR: Modality-Autoregressive World-Action Models

Adam Hung, Bardienus P. Duisterhof, Deva Ramanan, Jeffrey Ichnowski
Carnegie Mellon University
arXiv Code (coming soon)
ModAR: Predict the future one modality at a time, then act
1 Predicted tracks
motion
Predicted tracks Predicted tracks Predicted tracks Predicted tracks Predicted tracks Predicted tracks Predicted tracks Predicted tracks Predicted tracks
2 Predicted DINO
semantics
Predicted DINO Predicted DINO Predicted DINO Predicted DINO Predicted DINO Predicted DINO Predicted DINO Predicted DINO Predicted DINO
3 Predicted depth
geometry
Predicted depth Predicted depth Predicted depth Predicted depth Predicted depth Predicted depth Predicted depth Predicted depth Predicted depth
4 Predicted actions
live RGB
Predicted actions Predicted actions Predicted actions Predicted actions Predicted actions Predicted actions Predicted actions Predicted actions Predicted actions

Generating future tracks prediction

Real-world tasks
Simulation tasks
More real-world rollouts
1 Predicted tracks
motion
2 Predicted DINO
semantics
3 Predicted depth
geometry
4 Predicted actions
live RGB

1 / 8

Abstract

World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question. We introduce ModAR, the first WAM to autoregressively denoise multiple future modalities before predicting actions. This allows each prediction to condition on previously generated modalities. We train from scratch to systematically study how training-data mixtures, predicted modalities, and WAM formulations affect performance. In our evaluations, WAMs benefit from predicting point tracks, DINO features, and depth maps, while additionally predicting future RGB does not provide a consistent benefit. We also find that ModAR's sequential generation outperforms existing WAM formulations, with the highest average success rate at all evaluated data scales. We also fine-tune the video-model-initialized WAM Flex-π on the same data; ModAR achieves a slightly higher observed average success rate (75% vs. 72%) while using approximately 20× fewer training FLOPs and no pretraining. On three real-world bimanual tasks, ModAR outperforms baselines and improves with human videos.

ModAR compared to representative existing WAM formulations

Click a diagram to replay the animation

ModAR

 

obs Shared DiT backbone m₁ ··· mₖ actions

Future modalities are predicted sequentially, each conditioned on earlier predictions; actions are generated last.

Unified

DreamZero · Cosmos Policy

obs Shared DiT backbone m₁ ··· mₖ actions

All modalities and actions generated simultaneously with joint attention.

Disjoint

Fast-WAM

obs Shared DiT backbone m₁ ··· mₖ actions

All modalities and actions generated independently; future prediction is skipped at inference time.

Action-only

Diffusion Policy

obs DiT backbone actions

Actions generated directly. No future prediction.

Model Architecture

Click the diagram to replay the animation

Current observation DINO Depth RGB Global conditioning Robot configuration Task adaLN Shared cross-modality blocks ⋮ Modality-specific experts and heads DiT Modality-autoregressive generation Point Tracks Future DINO Future Depth Future RGB Robot Actions

Current DINO, depth, and RGB observations are embedded into tokens and processed by a shared diffusion transformer (DiT). The DiT combines shared cross-modality blocks with modality-specific experts and output heads. Robot configuration and a learned task embedding condition every DiT layer through adaLN. ModAR then denoises point tracks, DINO, depth, and RGB sequentially—each conditioned on earlier denoised modalities—and predicts robot actions last.

Results: Simulation

WAM Formulation Comparison

Average success rate over six RoboTwin tasks. All models are trained with 50 action-labeled demonstrations per task. We then add demonstrations with their action labels removed to increase the total demonstration count from 50 to 250 to 1,250. Independent-noise uses the same simultaneous generation as Unified at inference, but samples a separate flow timestep for each modality at each training step, following Flex-π. ModAR achieves the highest average success rate at every data scale and benefits most from the additional actionless data.

Predicted Modality Set Comparison

Average success rate for ModAR models trained with individual modalities and progressively larger modality sets, following the generation order tracks → DINO → depth → RGB. We find it beneficial to predict future tracks, DINO, and depth, while additionally predicting RGB provides no consistent gain.

Comparison to a video-model-initialized WAM

ModAR
Flex-π
Success rate
75%
72%
+3 points
Parameters
30.1M
6B
~200× smaller
Training FLOPs
1×
~20×
~20× less compute

We fine-tune Flex-π, initialized from the Wan2.2-TI2V-5B video model, on the same RoboTwin data at the 250-total-demonstration scale (50 action-labeled). ModAR reaches a slightly higher average success rate with approximately 200× fewer parameters, 20× fewer training FLOPs, and no pretraining.

Simulation Ablations

Average success rate over six RoboTwin tasks at the 250-demonstration scale (50 action-labeled). Context noise (noise added to previously generated modalities during training) is critical for robust autoregressive generation. Reversing the generation order to RGB → depth → DINO → tracks also reduces performance, supporting the choice to generate compact, structured modalities first. Removing tracks, DINO, or depth hurts more than removing RGB.

Results: Real-World Bimanual Manipulation

Real-World Training Data

Robot
100 demos
1 / 10
ID human
200 demos
1 / 10
EgoDex (OOD Human)
1,000 demos
1 / 10

Formulation Comparison

Real-world success rates with 30 trials per method per task. All methods use 100 robot demonstrations per task; ModAR and Unified additionally use 200 in-domain actionless human demonstrations and 1,000 out-of-domain EgoDex demonstrations. Real-world WAMs observe and predict tracks, DINO, and depth. ModAR achieves the highest success rate on all three tasks.

Learning from Actionless Human Data

We train separate ModAR models with just 100 robot demonstrations per task, then adding 200 in-domain actionless human demonstrations (used to supervise future-observation predictions, but not action prediction), and then also adding 1,000 out-of-domain human demonstrations from EgoDex (used similarly). The average success rate progressively improves from 70.0% to 81.1% to 83.3%.

Acknowledgements

We thank Narek Harutyunyan for assistance in data collection. This work is supported by the NSF GRFP (Grant Nos. DGE2140739 and DGE2631988). This work used Bridges-2 at Pittsburgh Supercomputing Center through ACCESS allocation CIS260202p.

BibTeX

@article{modar,
  title={Modality-Autoregressive World-Action Models},
  author={Hung, Adam and Duisterhof, Bardienus P. and Ramanan, Deva and Ichnowski, Jeffrey},
  year={2026}
}