Introducing SHAPE
SHAPE is a vision-language-action (VLA) model that learns dexterous manipulation from egocentric human recordings alongside robot teleoperation. Its backbone predicts discrete action tokens as an auxiliary task, and a shared flow-matching expert outputs two executable action streams, end-effector (EEF) and joint. A matched study on a bimanual robot with 22-DoF hands gives three findings.
01Discrete action supervision helps learn from human data
Robot demos0.47→0.80
Human demos only7→21/ 60
without → with discrete tokens
Predicting discrete tokens of each EEF chunk supervises the backbone directly. It drives the backbone's gradients and builds task-aligned action representations across human and robot data.
Why it helps02Two executable streams, each suited to a setting
Robot demos0.80vs0.52
Human demos only0vs21/ 60
joint vs EEF stream
One shared action expert outputs both streams, and either can drive the robot. Human pretraining improves the joint stream without any human joint labels.
Which stream to deploy03New tasks from human demonstrations alone
Robot demos0.80vs0.63
Human demos only21/ 60
SHAPE vs π0.5
Fine-tuned on human demonstrations alone, SHAPE learns tasks the robot was never teleoperated on. With robot demonstrations, it outperforms π0.5 and GR00T N1.7.
Results with human demosBackground
Human video is plentiful, but it carries no joint labels
Teleoperating two arms and two multi-fingered hands is expensive. Egocentric human recordings are far cheaper to collect at scale, and they promise two things: stronger pretraining, and new tasks learned from human demonstrations alone.
A multi-fingered robot hand is closer to the human hand than a gripper is, which narrows the embodiment gap but does not close it. Tracked human hands can be retargeted to end-effector (EEF) actions, the wrist poses and fingertip positions, but not to usable joint actions: inverse kinematics (IK) depends on the embodiment and yields arm postures outside the robot's own distribution.
Egocentric human recordings
2,811 hours · 73.1k episodes · 320 tasks
EEF actions, retargetedno joint actions
Robot teleoperation
228 hours · 10.5k episodes · 105 tasks
EEF actionsjoint actions
The two sources of pretraining data, from the head camera. Human clips play at 1×, robot clips at 2×.
The missing joint labels raise two questions.
Question 1 · Training
How should human data supervise the policy?
Most methods supervise EEF actions through a continuous decoder, so gradients reach the backbone only through it. Recent VLAs add discrete action tokens, but without isolating their effect: whether they help learn from human data has been unclear.
Question 2 · Deployment
How should a dexterous robot read out actions?
Multi-fingered hands are highly redundant: many joint configurations give the same fingertip positions. Earlier dual-head policies execute only the joint head. Yet joint readout is more accurate with robot demonstrations, and EEF readout through IK transfers better from human demonstrations.
Method
One backbone with discrete tokens, one expert with two streams
One backbone and one action expert serve human and robot data; only the supervised outputs differ. Switch views to see what is trained and what runs on the robot.
Human video
Robot teleoperationRobot observation
Discrete action tokensℒFAST
FAST tokens of the EEF chunk · auxiliary target
Vision-language backboneQwen3-VL-4B
reads images, the instruction and the current state
sub-task text ℒtextfuture frames ℒvideo
Shared action expertflow matching · 0.6BℒFM
EEF stream 55-d · wrist poses and fingertipsfor human-to-robot transfer
IKrobotthe two streams never attend to each other
Joint stream 65-d · arm and hand jointsfor fine-grained control
robotTraining: every sample gives the backbone a discrete action target. Human samples supervise the EEF stream; robot samples supervise both streams.Deployment: only the backbone and the action expert run. Each rollout executes one of the two streams, read from the same checkpoint.
Discrete action supervision
The backbone, a Qwen3-VL-4B, predicts FAST tokens of each EEF chunk as an auxiliary task. Human and robot chunks share one vocabulary, so every sample supervises the backbone directly rather than only through the action expert. The expert cannot read the tokens, so they can be dropped at deployment.
Dual-stream action expert
A 0.6B flow-matching transformer reads the backbone at every layer and denoises an EEF chunk and a joint chunk in parallel. The two streams never attend to each other, so either can run alone. Human samples supervise only the EEF stream, yet they shape the joint stream through the shared backbone and expert.
ℒ = ℒEEFFM + mJ ℒjointFM + 0.1 ℒFAST + ℒvideo + 0.2 ℒtext
mJ masks the samples without joint labels. Pretraining draws human and robot samples 1 : 1.
Deployment
The joint stream commands the joints directly. The EEF stream predicts wrist poses and fingertip positions, which inverse kinematics (IK) converts to joint targets. Both come from one checkpoint, so the stream can be chosen at deployment.
- BackboneQwen3-VL-4B
- Action expert0.6B, flow matching
- Action chunk64 steps at 60 Hz
- EEF stream55 dimensions
- Joint stream65 dimensions
- FAST vocabulary1,024 tokens
Full architecture and attention mask
Results at a glance
A matched 2×2 study isolates each ingredient
Four models are pretrained with or without human data and with or without discrete tokens; the architecture, the robot data and the training budget stay the same. All are fine-tuned and evaluated in the same way, next to two public baselines.
The grid shows the four models in each setting; the arrows show what each ingredient adds.
Platform and protocol
- Two 7-DoF arms and two 22-DoF dexterous hands, with a head camera and two wrist cameras.
- 20 trials per policy and task, with randomized object placements.
- Baselines π0.5 and GR00T N1.7 are fine-tuned on the same demonstrations.
Result 1
Fine-grained control with robot demonstrations
Six tasks have teleoperated demonstrations, from a precise insertion to in-hand twisting. With the joint stream, SHAPE is best or tied on every task. The final reinsertion of the two connectors, at millimeter tolerance, remains hard for every policy.
Per-task score, joint stream
Mean score over 20 trials; Reinsert USB and Reinsert Type-C earn stage credit. Click a task to watch it.
Where does the gain come from?
Not from the architecture alone: without human data, our model only matches π0.5. Discrete tokens raise the score in both streams, with or without human data, and human pretraining then lifts the joint stream further.
Six-task score of every policy
Mean of the six per-task scores, with 95% confidence intervals.
Full results table
Result 2
New tasks from human demonstrations alone
Three tasks have no robot demonstrations at all. Each model is fine-tuned on egocentric human demonstrations of the task only, then run on the robot.
Top: a human demonstration from the fine-tuning data. Below: autonomous rollouts of SHAPE with the EEF stream; the top view is turned by 180° to match the human's viewpoint.
Success with zero robot demonstrations of the task
Number of successful trials; EEF stream unless marked.
Either ingredient alone transfers little; combining them performs substantially best. Fine-tuning is identical for the four models, with the discrete-token loss off, so the difference comes from pretraining.
With the EEF stream, SHAPE reaches for the object wherever it is placed instead of replaying one trajectory. Most failures happen at the grasp: the hand closes too loosely to lift the object, a gap in morphology and kinematics that retargeting does not fully bridge.
Both streams of every model
Result 3
Which stream to deploy?
Both streams are read from the same fine-tuned checkpoint, so they differ in the readout alone. Which one works better depends on the fine-tuning data.
Fine-tuned on robot demonstrations
Direct joint readout is more precise
Offline, the two streams are equally accurate; the gap opens during execution. This is consistent with under-constrained IK: five fingertip targets do not determine a 22-DoF hand, so small errors can disturb the finger joints that set the grasp.
Fine-tuned on human demonstrations only
Wrist-frame EEF readout is what transfers
Retargeted human arm postures lie outside the robot's own distribution, so joint offsets learned around them fail on the robot. The EEF stream predicts hand motion relative to the wrist and leaves the joints to IK at the robot's actual posture.
Sharing the backbone and the expert also carries human data to the joint stream: human samples never supervise it in pretraining, yet they raise its score on the real robot and in simulation.
Result 4
A second robot, in simulation
Do the two ingredients also help on a robot that pretraining never saw? We fine-tune all models on the 24 GR-1 tabletop tasks of RoboCasa, which crosses two gaps at once: from real data to simulation, and to a humanoid with 6-DoF hands.
One successful rollout of SHAPE per task, joint stream. The percentage is SHAPE's success rate on that task.
Success rate on RoboCasa GR-1 tabletop tasks
Joint stream; 50 episodes per task, averaged over three checkpoints.
SHAPE leads both baselines overall, and without human data our model again only matches π0.5. The two ingredients add up almost independently, with human pretraining contributing the larger share.
Success rate on each of the 24 tasks
Analysis
Why discrete supervision helps
Two things remain to be explained: why human pretraining and discrete tokens work best together, and how human data improves the joint stream without joint labels. Four analyses trace both back to pretraining.
It generalizes once it is paired with diverse human data
On robot data alone, discrete tokens overfit: they fit the training episodes better but generalize worse to held-out tasks. Paired with diverse human recordings, they lower the open-loop position error on unseen human episodes and on every held-out robot split, including one read from the joint stream. Transfer from human EEF data to joint control thus already emerges in pretraining.
It makes human frames action-informative and close to robot frames of the same task
Without human pretraining data, the backbone carries almost no action signal on human frames. With it, discrete tokens strengthen that signal, and the effect reaches the joint expert, which human data never supervises. With no alignment loss, they also pull human frames toward robot frames of the same task, at every layer of the backbone.
It is the language model's main source of gradient, and the gradients agree
The token loss is the largest source of the language model's gradient at every checkpoint, and it does not conflict with flow matching. Gradients also align across the two streams and, at nearly every checkpoint, across human and robot batches, so human EEF supervision moves the shared parameters in directions that joint control also uses.
A strong gradient is not enough: predicting tokens matters too
Would any strong continuous action target on the backbone do as well? We replace the tokens with a regression head and rescale its loss so that its backbone gradient matches that of the tokens throughout pretraining. It lifts action probes even higher than the tokens do, but erodes the task structure shared across embodiments at the last layer.
Closed-loop performance follows the task structure, not the probe: the continuous head transfers less from human demonstrations (p = 0.055) and interferes with fine-tuning in simulation. Discrete tokens thus work best as a training target for the backbone: they sustain its gradient without reweighting and preserve task structure across embodiments.
Citation
@misc{li2026shapediscretesupervisionparallel,
title={SHAPE: Discrete Supervision and Parallel Continuous Streams for Learning Dexterous Manipulation from Egocentric Human Data},
author={Yaping Li and Zhihong Liu and Jiacheng Li and Yi Yang and Wenchao Yue and Mingzhe Ye and Xiangcheng Liu and Yiyang Chen and Haotian He and Qi Zhang and Haonan Wang and Dahua Lin and Zhijie Deng},
year={2026},
eprint={2611.24585},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2611.24585},
}
SHAPE