3PoinTr: 3D Point Tracks for Learning Manipulation from Unconstrained Human Videos

3PoinTr learns manipulation from unconstrained human videos: videos where the human demonstrator can act freely rather than mimicking target robot kinematics. 3PoinTr first predicts dense 3D point tracks — how the scene should move to complete the task — and then conditions a closed-loop multitask policy on these tracks. 3PoinTr outperforms strong behavior cloning and learning-from-video baselines across simulated and real-world evaluations.

Learning manipulation policies from human videos could greatly reduce the need for expensive robot demonstrations, but existing approaches typically require restrictive assumptions such as choreographed human motions, predefined keypoints, manual annotations, or known grasp locations. We propose 3PoinTr, a method for pretraining sample-efficient robot policies from unconstrained human videos by predicting dense 3D point tracks. In the unconstrained human demonstration videos, humans are free to follow whatever trajectories and manipulation strategies they see fit, rather than choreographing their motions to mimic a robot. 3PoinTr uses a lightweight visibility-aware transformer to learn how scene points should move from human videos, and then trains a closed-loop multitask robot policy to flexibly extract action-relevant priors from those predicted point tracks. With only 20 action-labeled robot demonstrations, 3PoinTr achieves a 25.0 percentage point higher average success rate than the strongest behavior cloning and video-pretraining baseline on real-world tasks, and a 29.6 percentage point higher average success rate in simulation. Targeted ablations support the key design choices and confirm the benefit of learning from actionless videos. We further show that 3PoinTr’s point track prediction transformer outperforms a strong baseline by preserving supervision over partially occluded points.

Diagram of the 3PoinTr network architecture. Given an initial point cloud, a transformer predicts dense 3D point tracks describing how objects should evolve during the task. A Perceiver-IO point track encoder compresses these tracks into a small set of learned point track tokens. The policy conditions on these point track tokens together with the current point cloud and robot state, enabling closed-loop action prediction. Beyond global conditioning, residual track-token cross-attention in every U-Net block gives the action head a direct path to the predicted object motion, encouraging a simpler mapping from point tracks to robot actions.

This work is supported by the NSF GRFP (Grant No. DGE2140739).