Contrastive Latent Action Learning from Human Videos for Robotic Manipulation
Weisheng Dai, Kai Lan, Jianyi Zhou, Xiu Su, Junwen Tong, Weili Guan, Bo Zhao, Shuo Yang
• arXiv (cs.RO)
ConLA is an unsupervised pretraining framework that extracts semantically consistent latent actions from unlabelled human demonstration videos without relying on explicit action tags. By utilizing contrastive disentanglement with action category and temporal priors, it isolates pure motion dynamics from complex visual background noise. For the first time, pretraining on human videos with ConLA surpasses the performance achieved by pretraining directly on real robot teleoperation trajectories.
Embodied AI
Vision-Language-Action
Latent Action Quantization
Contrastive Learning
Robotic Manipulation
Full Abstract
Contrastive Latent Action Learning from Human Videos for Robotic Manipulation
Authors: Weisheng Dai, Kai Lan, Jianyi Zhou, Xiu Su, Junwen Tong, Weili Guan, Bo Zhao, Shuo Yang
Journal: arXiv (cs.RO)
Published: January 2026
Overcoming Shortcut Learning in Unsupervised Video Pretraining
Pretraining Vision-Language-Action (VLA) models on human video demonstrations provides scalable motion priors, but existing VQ-VAE frameworks suffer from shortcut learning. Because optimization is based purely on visual reconstruction, models memorize background appearance rather than learning true inter-frame dynamics, resulting in entangled action spaces.
ConLA introduces an unsupervised contrastive disentanglement framework that isolates motion dynamics from visual noise by enforcing semantic action priors and temporal sensitivity.
The Three-Stage ConLA Pipeline
Contrastive Latent Action Learning: Uses a spatial-temporal Transformer as an Inverse Dynamics Model (IDM) encoder, splitting latent representations into dedicated action and visual heads:
Action-Centric Contrastive Loss: Employs supervised contrastive learning to pull identical action categories together into compact clusters across diverse environments.
Vision-Centric Contrastive Loss: Applies an InfoNCE loss via temporal inverse augmentation ([Ot+k, Ot]) to isolate motion-invariant appearance features from dynamic changes.
Latent Action Policy Pretraining: Trains an autoregressive Large World Model (LWM-7B) to predict discrete latent action tokens conditioned on video frames and text instructions, acting as pseudo-labels.
Action Fine-Tuning: Maps the learned latent action tokens to executable motor control signals using a minimal set of ground-truth teleoperation trajectories.
Empirical Validation & Benchmark Performance
SimplerEnv Simulation: ConLA achieves a 64.6% average success rate when pretrained solely on human videos (Something-Something V2), outperforming the baseline LAPA (52.1%) by a massive +12.5% margin.
Beating Real Robot Pretraining: In a landmark result, human-video pretraining with ConLA (64.6%) surpassed the model pretrained directly on real robot trajectories (ActionVLA at 63.5%).
Real-World Deployment: On a Franka Research 3 arm, ConLA achieved 48.2% average success in real-world tabletop tasks under unseen object combinations, outperforming LAPA (32.3%) by +15.9%.
One-Line Takeaway: ConLA unlocks large-scale internet human videos for VLA pretraining by successfully disentangling pure motion dynamics from complex visual noise, producing robust robot policies.