Head2Wrist: Wrist-Camera Robot Policies from Human Head-Camera Demonstrations without a Scene Scan

Abstract

Human head-camera demonstrations make up most large open manipulation datasets, but a robot trained on them meets two gaps: a moving human head rarely reproduces the robot's head view, and the arm in the images is human; inpainting a robot arm over it keeps the first. We instead train on the robot's wrist camera, whose view rides on the gripper, and generate it from a single human head-camera demonstration (posed head video and hand tracking), with no separate scene scan and no teleoperation. We reconstruct the scene from the pre-grasp head frame and nearby posed frames, plan a robot-executable, hand-aligned trajectory checked in a physics twin, and render the wrist views with layers matched to the robot's camera (dynamic object, gripper, robot body, cast shadows). In closed loop on the real FFW-SG2, a policy trained only on views generated from three Meta Quest 3 demonstrations (a cube into a bowl) succeeds in 18 of 20 trials from cube starts within 3 cm and 30° of a demonstrated one, where replaying the planned path open loop succeeds in 5. In a matched training-image comparison in Isaac Lab twins of six EgoDex scenes, with the generator given a clean twin head render and the object's exact geometry and start pose, ACT policies trained only on generated views show no detectable difference from true-view policies on three development scenes (all 9 paired p ≥ 0.12); on two held-out scenes they succeed less often in range (47/60 vs 58/60) and far less outside it (4/40 vs 27/40), and on a 1.4 cm-thin phone they succeed in 3 of 30 in-range trials (true view: 15).

Method

The pipeline has four stages:

1. Scene from the head video. Inpaint the hand in the pre-grasp head frame and estimate depth with Depth Anything 3 on nearby posed frames. Match the table plane to the metric hand track with one scale factor, fit the table, segment objects with SAM 3, and lift the grasped object to a depth-refined SAM 3D mesh. The scene proxy extends the table beyond the head view.
2. Robot plan and physics check. Plan an FFW-SG2 tool-center-point path aligned with the human pinch and palm: overview, descent, grasp, lift, transport, set-down and retreat. Search base position, heading and lift for a stance from which arm-only inverse kinematics reaches the path. An Isaac Lab twin replays the motion with physics and tries grasp variants; demonstrations that cannot pick and place are rejected.
3. Layered wrist-view generator. Render the scene proxy along the planned D405 camera path using visible head-frame pixels and an outpainted table map. Composite the moving grasped object, the URDF gripper at its planned opening with target-camera textures, the robot arm and body, and cast shadows from randomized lighting.
4. Policy. Train ACT on the active arm's 424 × 240 wrist images with an end-effector hybrid representation: absolute TCP pose and jaw width as state, and TCP poses relative to the current pose plus absolute jaw width as actions, executed through inverse kinematics. Re-plan for new object starts and robot stances; train with stance randomization and strong affine, colour, blur and noise augmentation.

Real-Robot Results

Three of four Quest 3 cube-into-bowl demonstrations passed the physics check. Re-planning for object starts within ±8 cm and ±30° and randomized stances produced 252 training episodes. On 20 test starts inside that range, within 3 cm and 30° of a demonstrated start, the policy trained only on generated views succeeded in 18/20 trials; open-loop replay of the planned path succeeded in 5/20. Both policy failures released the cube outside the bowl. This comparison tests closed-loop use near demonstrated starts; it does not establish an advantage over other training data.

Simulation: A Matched Training-Image Comparison

The two ACT policies use the same trajectories, states and actions; only their training wrist images differ. A scripted expert in each Isaac Lab twin provides 225–630 demonstrations. The generator replaces every wrist frame using one clean twin head render and the object's exact geometry and start pose. Both policies are evaluated closed loop on the true simulated D405 from paired starts. These controlled inputs do not test reconstruction from raw human video.

In-range starts use the training range (±8 cm, ±30°). Out-of-range starts extend up to 12 cm with at least one axis outside that range. Moved-base tests use in-range object starts with the base moved by up to 5 cm and 10°. Success requires lifting the object, an open gripper at the end, and a final object position within 3 cm in XY and 2 cm in height of the expert's placement.

Counts transcribe Figure 3; its intervals are 95% Wilson intervals. * Paired exact McNemar p < 0.05; stack/18 out of range pools 16 + 24 starts. On stack/18, stack/48 and pick-place/25, none of the nine paired comparisons differs significantly (all p ≥ 0.12). No-difference results are not equivalence tests. A second training seed on stack/48 and pick-place/25 also gives no significant paired difference (all six p ≥ 0.25; between seeds all p ≥ 0.12).

A gap remains on the held-out scenes. With generator and recipe frozen, generated-view success is lower in all six held-out test sets: pooled generated versus true counts are 47/60 versus 58/60 in range, 4/40 versus 27/40 out of range, and 25/48 versus 35/48 with a moved base (exact p = 0.003, < 10⁻⁴, 0.02). On stack/48 out of range, both policies are far below the expert (5/20 and 6/20 versus 16/20).

The 1.4 cm-thin phone is a failure case. On pick-place/1, generated versus true views produce 3/30 versus 15/30 in-range successes and 18 versus 10 rollouts with no lift.

Ablations

Representation, stance randomization and strong augmentation affect success. Object-pose and size errors reduce transfer, and bundled generator revisions do not isolate single-change effects. Table 1 reports closed-loop tests on the true camera.

EEF absolute uses absolute TCP-pose actions. ‡ Per-episode error in the generator's object start pose and size; median scale errors are 2.5% / 4.9%. † Constant grey wrist image in training and test on stack/48 + pick-place/25; stopped early.

Base: scene, object and gripper. Colour: colour-matched table extension and target-camera gripper texture. Seamless: seamless extension. Body: robot body and shadows. Revisions were chosen on development scenes by inspection and teacher-forced image-region swaps; on stack/18 out of range, success went from 33/40 (base generator) to 29/40 (full).

What Did Not Work

The generator must follow a close-range camera and keep the scene. In pilots, Lyra 2.0, from the hand-removed head frame, drifted from the commanded camera near the table and GEN3C dropped objects once the wrist camera magnified the head frame; over the scene proxy, Cosmos 3 repainted the scene and VACE inserted reference objects. Cosmos-Transfer2.5 keeps the scene, but refining the generated views (27 GPU-min per episode) gave no gain on stack/48 at 20 demonstrations (refined versus ours: 21/30 versus 20/30 in range, 12/24 versus 13/24 moved base).

Limitations

In simulation the generator receives the object's geometry and start pose from the twin; in the real pipeline these come from the head frame and SAM 3D. Synthetic object errors lower success. Four scenes served development and generator revisions were chosen on them, which may partly explain the held-out gap; the thin phone remains a failure case. No-difference results are not equivalence tests. Two main comparisons have a second seed, and bundled generator revisions hide single-change effects. Real trials cover one task near its demonstrated start and, against open-loop replay, show closed-loop use, not an edge over other training data.

BibTeX

@inproceedings{kim2026head2wrist,
  title={Head2Wrist: Wrist-Camera Robot Policies from Human Head-Camera Demonstrations without a Scene Scan},
  author={Kim, Juno and Park, Yesol and Yoon, Hye-Jung and Zhang, Byoung-Tak},
  booktitle={CoRL Workshop on UMI Arena},
  year={2026}
}

License

The code, model, dataset, and raw demonstrations are released under the MIT License.