Contrastive Latent Action Learning from Human Videos for Robotic Manipulation

Overcoming Shortcut Learning in Unsupervised Video Pretraining

Pretraining Vision-Language-Action (VLA) models on human video demonstrations provides scalable motion priors, but existing VQ-VAE frameworks suffer from shortcut learning. Because optimization is based purely on visual reconstruction, models memorize background appearance rather than learning true inter-frame dynamics, resulting in entangled action spaces.

ConLA introduces an unsupervised contrastive disentanglement framework that isolates motion dynamics from visual noise by enforcing semantic action priors and temporal sensitivity.

The Three-Stage ConLA Pipeline

1. Contrastive Latent Action Learning: Uses a spatial-temporal Transformer as an Inverse Dynamics Model (IDM) encoder, splitting latent representations into dedicated action and visual heads:
   • Action-Centric Contrastive Loss: Employs supervised contrastive learning to pull identical action categories together into compact clusters across diverse environments.
   • Vision-Centric Contrastive Loss: Applies an InfoNCE loss via temporal inverse augmentation ([O_t+k, O_t]) to isolate motion-invariant appearance features from dynamic changes.
2. Latent Action Policy Pretraining: Trains an autoregressive Large World Model (LWM-7B) to predict discrete latent action tokens conditioned on video frames and text instructions, acting as pseudo-labels.
3. Action Fine-Tuning: Maps the learned latent action tokens to executable motor control signals using a minimal set of ground-truth teleoperation trajectories.

Empirical Validation & Benchmark Performance

• SimplerEnv Simulation: ConLA achieves a 64.6% average success rate when pretrained solely on human videos (Something-Something V2), outperforming the baseline LAPA (52.1%) by a massive +12.5% margin.
• Beating Real Robot Pretraining: In a landmark result, human-video pretraining with ConLA (64.6%) surpassed the model pretrained directly on real robot trajectories (ActionVLA at 63.5%).
• Real-World Deployment: On a Franka Research 3 arm, ConLA achieved 48.2% average success in real-world tabletop tasks under unseen object combinations, outperforming LAPA (32.3%) by +15.9%.

One-Line Takeaway: ConLA unlocks large-scale internet human videos for VLA pretraining by successfully disentangling pure motion dynamics from complex visual noise, producing robust robot policies.