Skip to content

From a captured space to a capable robot

OffSite

Mobile Manipulation from a Single Scene Capture,
without On-Site Teleoperation

Capture the scene once. Generate robot demonstrations off site. Deploy a policy that navigates, grasps, and carries in the real space.

Full workflow · 2:41 Robot demos

The complete workflow / 2:41

From scene capture
to real-world deployment.

See the full process: handheld capture, scene and task setup, verified simulation, generated training views, and the deployed robot. Use the chapters to jump to a stage.

On the real robot

AI-Worker FFW-SG2

Task 04 Real deployment

Cross-Room Delivery

Pick up the object, turn, navigate 3–4 m across the room, and place it at the sink.

Pick + navigate + place

14/30 real-robot successes

Trained on 300 generated demonstrations.

Deployed policy · Real head-camera view 4× playback

Four tasks, each in a different captured environment. These clips show the deployed policy; the full workflow video also pairs generated training views with real execution.

13 min
Average human effort per scene
7 min capture + 6 min layout
300
Generated demonstrations per task
No on-site teleoperation
60%
Pooled real-robot success
72 successes / 120 trials

01 / The scene becomes the training ground

One walkthrough.
Many places to act.

A captured room carries more than its appearance. It supplies the geometry a mobile robot must navigate and the surfaces its arms must interact with.

OffSite turns that capture into a reusable environment for generating demonstrations, with different task sites, object placements, and robot starting configurations.

A single reconstructed room supports task sites in the kitchen, dining area, bookshelf, and living room, with real and generated robot-camera views shown alongside.
One capture can support multiple task sites. The four-task evaluation uses four different captured scenes.

02 / How it works

Capture. Verify. Learn.

Appearance and contact geometry come from the same reconstruction, so camera views and physical interactions share one coordinate frame.

  1. 01

    Reconstruct the scene

    A 3D Gaussian Splatting model supplies scene appearance. A mesh from the same reconstruction supplies contact and collision geometry.

  2. 02

    Generate verified rollouts

    Sample task layouts, execute grasp and base motions in simulation, and keep only collision-free, reachable, successful episodes. Vision feedback refines grasp parameters after failures.

  3. 03

    Train, then deploy

    Depth compositing creates head- and wrist-camera training views. A vision-language-action policy learns from those views, robot state, and instructions, then runs closed loop on the real robot.

OffSite pipeline: scene capture yields an appearance model and physics mesh, verified simulation rollouts become synchronized camera observations, and the trained policy is deployed on the real robot.
One scene representation connects capture, verified demonstration generation, policy training, and real-world deployment.

03 / Real-robot evidence

Scale the data.
Keep the on-site effort fixed.

At 90 demonstrations per task, generated data and on-site teleoperation both reach 43.3% pooled success. Scaling generated data to 300 demonstrations per task raises success to 60.0%, with the same one-time capture and layout effort.

Scroll to compare all results

Real-robot success (%), 30 trials per task. Column labels give the training demonstrations per task.
Task Teleoperation
90 demos 		OffSite
90 demos 		OffSite
300 demos
Navigate-to-Pick 56.7 56.7 70.0
Vertical Shelf Transfer 50.0 53.3 73.3
Bimanual Transport 36.7 30.0 50.0
Cross-Room Delivery 30.0 33.3 46.7
Pooled 43.3 43.3 60.0

What the result depends on

Capture the place
the robot will work in.

The walkthrough must cover the viewpoints the robot will occupy. Missing views or geometry can reduce the yield of usable demonstrations, and substantial changes to the scene require a fresh capture.

The evaluation covers one robot platform and four rigid-object tasks. Deformable manipulation, tactile dexterity, and transfer across robot embodiments are outside this evaluation.