SHAPE

Supervision from Human Action tokens with Parallel Executable streams

Discrete supervision and parallel continuous streams for learning dexterous manipulation from egocentric human data.

 

Introducing SHAPE

SHAPE is a vision-language-action (VLA) model that learns dexterous manipulation from egocentric human recordings alongside robot teleoperation. Its backbone predicts discrete action tokens as an auxiliary task, and a shared flow-matching expert outputs two executable action streams, end-effector (EEF) and joint. A matched study on a bimanual robot with 22-DoF hands gives three findings.

01Discrete action supervision helps learn from human data

Robot demos0.470.80

Human demos only721/ 60

without → with discrete tokens

Predicting discrete tokens of each EEF chunk supervises the backbone directly. It drives the backbone's gradients and builds task-aligned action representations across human and robot data.

Why it helps

02Two executable streams, each suited to a setting

Robot demos0.80vs0.52

Human demos only0vs21/ 60

joint vs EEF stream

One shared action expert outputs both streams, and either can drive the robot. Human pretraining improves the joint stream without any human joint labels.

Which stream to deploy

03New tasks from human demonstrations alone

Robot demos0.80vs0.63

Human demos only21/ 60

SHAPE vs π0.5

Fine-tuned on human demonstrations alone, SHAPE learns tasks the robot was never teleoperated on. With robot demonstrations, it outperforms π0.5 and GR00T N1.7.

Results with human demos

Background

Human video is plentiful, but it carries no joint labels

Teleoperating two arms and two multi-fingered hands is expensive. Egocentric human recordings are far cheaper to collect at scale, and they promise two things: stronger pretraining, and new tasks learned from human demonstrations alone.

A multi-fingered robot hand is closer to the human hand than a gripper is, which narrows the embodiment gap but does not close it. Tracked human hands can be retargeted to end-effector (EEF) actions, the wrist poses and fingertip positions, but not to usable joint actions: inverse kinematics (IK) depends on the embodiment and yields arm postures outside the robot's own distribution.

Egocentric human recordings

2,811 hours · 73.1k episodes · 320 tasks

EEF actions, retargetedno joint actions

Robot teleoperation

228 hours · 10.5k episodes · 105 tasks

EEF actionsjoint actions

The two sources of pretraining data, from the head camera. Human clips play at 1×, robot clips at 2×.

The missing joint labels raise two questions.

Question 1 · Training

How should human data supervise the policy?

Most methods supervise EEF actions through a continuous decoder, so gradients reach the backbone only through it. Recent VLAs add discrete action tokens, but without isolating their effect: whether they help learn from human data has been unclear.

Question 2 · Deployment

How should a dexterous robot read out actions?

Multi-fingered hands are highly redundant: many joint configurations give the same fingertip positions. Earlier dual-head policies execute only the joint head. Yet joint readout is more accurate with robot demonstrations, and EEF readout through IK transfers better from human demonstrations.


Method

One backbone with discrete tokens, one expert with two streams

One backbone and one action expert serve human and robot data; only the supervised outputs differ. Switch views to see what is trained and what runs on the robot.

supervised by human datasupervised by robot datatraining only

Human video

head camera
EEF actions

Robot teleoperationRobot observation

head camera
left wrist
right wrist
EEF actions
joint actions

Discrete action tokensℒFAST

FAST tokens of the EEF chunk · auxiliary target

Vision-language backboneQwen3-VL-4B

reads images, the instruction and the current state

sub-task text ℒtextfuture frames ℒvideo

Shared action expertflow matching · 0.6BℒFM

EEF stream 55-d · wrist poses and fingertipsfor human-to-robot transfer

IKrobot

the two streams never attend to each other

Joint stream 65-d · arm and hand jointsfor fine-grained control

robot

Training: every sample gives the backbone a discrete action target. Human samples supervise the EEF stream; robot samples supervise both streams.Deployment: only the backbone and the action expert run. Each rollout executes one of the two streams, read from the same checkpoint.

Discrete action supervision

The backbone, a Qwen3-VL-4B, predicts FAST tokens of each EEF chunk as an auxiliary task. Human and robot chunks share one vocabulary, so every sample supervises the backbone directly rather than only through the action expert. The expert cannot read the tokens, so they can be dropped at deployment.

Dual-stream action expert

A 0.6B flow-matching transformer reads the backbone at every layer and denoises an EEF chunk and a joint chunk in parallel. The two streams never attend to each other, so either can run alone. Human samples supervise only the EEF stream, yet they shape the joint stream through the shared backbone and expert.

ℒ = ℒEEFFM + mJ ℒjointFM + 0.1 ℒFAST + ℒvideo + 0.2 ℒtext

mJ masks the samples without joint labels. Pretraining draws human and robot samples 1 : 1.

Deployment

The joint stream commands the joints directly. The EEF stream predicts wrist poses and fingertip positions, which inverse kinematics (IK) converts to joint targets. Both come from one checkpoint, so the stream can be chosen at deployment.

  • BackboneQwen3-VL-4B
  • Action expert0.6B, flow matching
  • Action chunk64 steps at 60 Hz
  • EEF stream55 dimensions
  • Joint stream65 dimensions
  • FAST vocabulary1,024 tokens
Full architecture and attention mask
Overview of SHAPE: inputs, backbone, flow-matching action expert, training losses, deployment and attention mask.
Human recordings (orange) provide head images and EEF actions; robot data (blue) add wrist images and joint actions. Dashed parts are used in training only. The attention mask keeps the EEF and joint streams apart and hides the FAST tokens from the action expert.

Results at a glance

A matched 2×2 study isolates each ingredient

Four models are pretrained with or without human data and with or without discrete tokens; the architecture, the robot data and the training budget stay the same. All are fine-tuned and evaluated in the same way, next to two public baselines.

0.80six-task score with robot demos (π0.5: 0.63)
21/ 60trials on three new tasks, from human demos alone
72.2%success on a second robot in simulation (π0.5: 64.8%)
1,680real-robot trials

The grid shows the four models in each setting; the arrows show what each ingredient adds.

The robot seen from the side: head, torso and two arms with five-fingered hands.

Platform and protocol

  • Two 7-DoF arms and two 22-DoF dexterous hands, with a head camera and two wrist cameras.
  • 20 trials per policy and task, with randomized object placements.
  • Baselines π0.5 and GR00T N1.7 are fine-tuned on the same demonstrations.

Result 1

Fine-grained control with robot demonstrations

Six tasks have teleoperated demonstrations, from a precise insertion to in-hand twisting. With the joint stream, SHAPE is best or tied on every task. The final reinsertion of the two connectors, at millimeter tolerance, remains hard for every policy.

Per-task score, joint stream

Mean score over 20 trials; Reinsert USB and Reinsert Type-C earn stage credit. Click a task to watch it.

Where does the gain come from?

Not from the architecture alone: without human data, our model only matches π0.5. Discrete tokens raise the score in both streams, with or without human data, and human pretraining then lifts the joint stream further.

Six-task score of every policy

Mean of the six per-task scores, with 95% confidence intervals.

Full results table

Result 2

New tasks from human demonstrations alone

Three tasks have no robot demonstrations at all. Each model is fine-tuned on egocentric human demonstrations of the task only, then run on the robot.

Top: a human demonstration from the fine-tuning data. Below: autonomous rollouts of SHAPE with the EEF stream; the top view is turned by 180° to match the human's viewpoint.

Success with zero robot demonstrations of the task

Number of successful trials; EEF stream unless marked.

Either ingredient alone transfers little; combining them performs substantially best. Fine-tuning is identical for the four models, with the discrete-token loss off, so the difference comes from pretraining.

With the EEF stream, SHAPE reaches for the object wherever it is placed instead of replaying one trajectory. Most failures happen at the grasp: the hand closes too loosely to lift the object, a gap in morphology and kinematics that retargeting does not fully bridge.

Both streams of every model

Result 3

Which stream to deploy?

Both streams are read from the same fine-tuned checkpoint, so they differ in the readout alone. Which one works better depends on the fine-tuning data.

Fine-tuned on robot demonstrations

Direct joint readout is more precise

Offline, the two streams are equally accurate; the gap opens during execution. This is consistent with under-constrained IK: five fingertip targets do not determine a 22-DoF hand, so small errors can disturb the finger joints that set the grasp.

Fine-tuned on human demonstrations only

Wrist-frame EEF readout is what transfers

Retargeted human arm postures lie outside the robot's own distribution, so joint offsets learned around them fail on the robot. The EEF stream predicts hand motion relative to the wrist and leaves the joints to IK at the robot's actual posture.

Sharing the backbone and the expert also carries human data to the joint stream: human samples never supervise it in pretraining, yet they raise its score on the real robot and in simulation.


Result 4

A second robot, in simulation

Do the two ingredients also help on a robot that pretraining never saw? We fine-tune all models on the 24 GR-1 tabletop tasks of RoboCasa, which crosses two gaps at once: from real data to simulation, and to a humanoid with 6-DoF hands.

One successful rollout of SHAPE per task, joint stream. The percentage is SHAPE's success rate on that task.

Success rate on RoboCasa GR-1 tabletop tasks

Joint stream; 50 episodes per task, averaged over three checkpoints.

SHAPE leads both baselines overall, and without human data our model again only matches π0.5. The two ingredients add up almost independently, with human pretraining contributing the larger share.

Success rate on each of the 24 tasks

Analysis

Why discrete supervision helps

Two things remain to be explained: why human pretraining and discrete tokens work best together, and how human data improves the joint stream without joint labels. Four analyses trace both back to pretraining.

It generalizes once it is paired with diverse human data

On robot data alone, discrete tokens overfit: they fit the training episodes better but generalize worse to held-out tasks. Paired with diverse human recordings, they lower the open-loop position error on unseen human episodes and on every held-out robot split, including one read from the joint stream. Transfer from human EEF data to joint control thus already emerges in pretraining.

Offline right-wrist error across pretraining steps for the four matched models.
(a) Drop in right-wrist error from adding discrete tokens to pretraining with human data, by split, with 95% confidence bands. (b) Right-wrist error of the four models during pretraining.

It makes human frames action-informative and close to robot frames of the same task

0.143probe R² on human frames (0.089 without tokens)
0.281robot-to-human probe transfer (0.205 without)
57%same-task robot neighbors (37% without, 33% by chance)

Without human pretraining data, the backbone carries almost no action signal on human frames. With it, discrete tokens strengthen that signal, and the effect reaches the joint expert, which human data never supervises. With no alignment loss, they also pull human frames toward robot frames of the same task, at every layer of the backbone.

Linear probe scores of the four matched models, and the share of same-task robot neighbors at the last layer and across the layers of the backbone.
(a) Linear action probes on last-layer features at 30k steps. (b) Share of a human frame's ten nearest robot frames that come from its own task. (c) The same share across the layers of the backbone.

It is the language model's main source of gradient, and the gradients agree

70%of the language model's gradient on human batches comes from the token loss
93%on robot batches
+0.84cosine between EEF and joint gradients on the shared expert

The token loss is the largest source of the language model's gradient at every checkpoint, and it does not conflict with flow matching. Gradients also align across the two streams and, at nearly every checkpoint, across human and robot batches, so human EEF supervision moves the shared parameters in directions that joint control also uses.

Gradient-energy share per module, and cosines between the gradients of losses, streams and data sources across pretraining.
(a) Share of each module's gradient energy by loss at 30k steps, on human (H) and robot (R) batches. (b–d) Cosines between mean gradients: tokens against flow matching, EEF against joint stream, human against robot batches.

A strong gradient is not enough: predicting tokens matters too

Would any strong continuous action target on the backbone do as well? We replace the tokens with a regression head and rescale its loss so that its backbone gradient matches that of the tokens throughout pretraining. It lifts action probes even higher than the tokens do, but erodes the task structure shared across embodiments at the last layer.

Bar charts of the model without discrete tokens, the continuous-head control and SHAPE: last-layer probe, same-task neighbors, and success on the new tasks.
Last-layer probe R², same-task neighbors (dashed: chance) and success on the new tasks, with 95% intervals.

Closed-loop performance follows the task structure, not the probe: the continuous head transfers less from human demonstrations (p = 0.055) and interferes with fine-tuning in simulation. Discrete tokens thus work best as a training target for the backbone: they sustain its gradient without reweighting and preserve task structure across embodiments.


Citation

@misc{li2026shapediscretesupervisionparallel,
      title={SHAPE: Discrete Supervision and Parallel Continuous Streams for Learning Dexterous Manipulation from Egocentric Human Data},
      author={Yaping Li and Zhihong Liu and Jiacheng Li and Yi Yang and Wenchao Yue and Mingzhe Ye and Xiangcheng Liu and Yiyang Chen and Haotian He and Qi Zhang and Haonan Wang and Dahua Lin and Zhijie Deng},
      year={2026},
      eprint={2611.24585},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2611.24585},
}

BibTeX on arXiv