3PoinTr: From Human Videos to Robot Policies with 3D Point-Track Plans

3PoinTr learns from human videos by treating 3D point tracks as global plans for robot policies. From an initial observation, it predicts complete-task point tracks and compresses them into a few plan tokens that guide a closed-loop policy throughout execution. Across eight simulation and real-world tasks with 20 robot demonstrations, 3PoinTr achieves a 79.3% average success rate, compared with 52.0% for the strongest behavior-cloning or learning-from-video baseline.

Human Video Pretraining

➔

Downstream Policy Rollouts

Abstract

Human videos could reduce reliance on expensive robot demonstrations, but extracting task-relevant features remains challenging. We introduce 3PoinTr, a method that provides robot policies with a global 3D point-track plan predicted from human videos. Given an initial point cloud, 3PoinTr predicts dense 3D point tracks spanning the full task and compresses them into a small set of tokens that remain fixed throughout execution. At each policy query, the policy combines these plan tokens with the current observation to predict robot actions. This formulation transfers exceptionally well even when human demonstrations use different viable motions from the robot. With 20 robot demonstrations, 3PoinTr improves average success over the strongest behavior-cloning or video-pretraining baseline by 25.0 percentage points on real-world tasks and 29.6 percentage points in simulation. Ablations establish the benefits of predicting non-embodiment 3D tracks once as a global plan, Perceiver-IO compression, and residual cross-attention. The point-track predictor also outperforms a strong 3D-flow baseline by retaining supervision for temporarily occluded points.

Model Architecture

Given the initial point cloud, a transformer predicts dense 3D point tracks spanning the full task. A Perceiver-IO encoder compresses these tracks into a small set of plan tokens that are cached and held fixed throughout the episode. At each action step, the policy combines these plan tokens with the current point cloud and robot state. Residual track-token cross-attention gives the flow-matching action head a direct path to the predicted object motion.

Results: Sample-Efficient Policy Learning

Simulation Tasks

Simulation success rates (%) evaluated over 200 rollouts per task. Results are reported for policies trained with 20, 50, and 100 action-labeled demonstrations and 100 actionless videos per task. 3PoinTr achieves the highest average success rate across all numbers of demonstrations.

Real-World Tasks

Real-world success rates for policies trained with 20 robot demonstrations and 50 human videos per task, evaluated over 20 rollouts. 3PoinTr achieves the highest success rate on all four tasks, with a 25.0 percentage point higher average success rate than the best baseline, DP3.

Ablations

Simulation Ablations

Ablation success rates (%) evaluated over 200 rollouts per simulation task, for policies trained with 20 demonstrations. 3PoinTr achieves the best average success rate, demonstrating the value of 3D point tracks, the Perceiver IO encoder, U-Net cross-attention, and additional actionless videos.

Real-World Ablations

Real-world ablation success rates over 20 rollouts per task, for policies trained with 20 robot demonstrations and 50 human videos. Predicting non-embodiment tracks once from the initial observation outperforms both including demonstrator embodiment motion and re-predicting tracks at every policy query.

Rollouts

3D Point Track Prediction

Example Policy Rollouts

Results: 3D Point Track Prediction

We also evaluate the quality of 3PoinTr's 3D point track predictions. Here we show Average Distance Error (ADE) and ADE of the 5% of points that move the most (5% ADE) for 3PoinTr and General Flow on real-world tasks (in millimeters). 3PoinTr outperforms General Flow in both metrics on every task, with average error reductions of 28.0% and 44.1% compared to General Flow.