APT: Action Expert Pretraining Improves Instruction
Generalization of Vision-Language-Action Policies

Vision-Language-Action (VLA) models that couple pretrained VLMs with continuous action experts achieve strong manipulation, yet generalize poorly to out-of-distribution (OOD) language instructions. The cause is a structural imbalance in VLA data — language is far less diverse than visual and action content — which biases policies toward visual shortcuts. Unlike discrete-action methods protected by vision-language co-training, continuous action experts learn from scratch on this imbalanced data, producing noisy gradients that corrupt the VLM. From a Bayesian perspective, we factorize the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage method emphasizing Action expert PreTraining: Stage 1 pretrains the action expert as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance; Stage 2 injects language through a gated fusion mechanism that preserves the learned visuomotor prior. APT applies to mainstream VLA architectures (π- and GR00T-style) and consistently improves generalization to unseen instructions and compositional tasks.

We factorize the VLA policy from a Bayesian perspective, separating action generation into a language-agnostic Vision-Action (VA) prior \(\pi^{p}(\mathbf{a}\mid\mathbf{v})\) and a language-conditioned VLA likelihood \(L(\ell\mid\mathbf{v},\mathbf{a})\). The key observation is that although full VLA triplets suffer from language-vision imbalance, vision-action pairs alone are well-balanced and create no shortcut incentive.

Stage 1 — VA Prior. A diffusion-based action expert is pretrained as a VA prior conditioned solely on visual tokens from a frozen VLM backbone, with language tokens masked from all self-attention computations — bypassing the language imbalance and providing a principled initialization before any language is introduced. Stage 2 — VLA Likelihood. Language tokens are injected through newly introduced interleaved attention layers, and all layers are jointly finetuned starting from the Stage 1 checkpoint. A novel action expert design with a layer-wise gated fusion mechanism integrates VLM features into the action expert, inheriting the VLM's representational capacity while preserving the pretrained visuomotor prior. The two-stage scheme applies to mainstream continuous-action VLA architectures, such as the π- and GR00T-style architectures.