Two tasks, one human video each (serving bread, watering flowers), on a robot twin: dual Tianji 7-DoF arms, two Sharpa Wave 22-DoF hands, a ZED camera. Every rollout ran in Isaac Sim; this page replays the recorded states.
Studio = seamless floor at the table-top height · Lab = the Isaac table. Drag to orbit (turns the auto camera off), scroll to zoom; keys: space, ← →, 1–4 views, c, o, f.
One human video, four ways to use it, and the twin's own ceiling. Every arm runs the same 30 randomized bread and plate placements of serve_bread_002 in Isaac Sim. Original physics: the cem8 controller, the bread about 2.7× too heavy, release triggered by the plate pose. Realistic physics: a 75 g croissant with measured friction and 20 N fingertip caps, the policy releases on its own, each placement run with two seeds. The numbers are counted in your browser from the recorded rollouts below.
The fixed evaluation of every arm, from the raw rollout files. Strict success: the task done and held (bread at rest on the plate for 1 s after an open-hand release; ≥ 100 ml on the flowers with 1 s of on-target water and the pot intact). Subtask reach: the share of the 60 rollouts (30 placements × 2 seeds) that got at least that far, every earlier subtask reached too. Completion score: the subtasks reached, each counted once reached, equal weights, approach not scored. Intervals: Wilson 95 % for success, placement-cluster bootstrap 95 % for completion.
Every robot-twin dataset generated from the human videos, published on Hugging Face: each episode's result json, the per-row recording (_viz.npz: joint angles, object and hand poses, fingertips, camera intrinsics and extrinsics) and the policy-input video. The policies on this page were finetuned on their strict successes; failures are kept so the data can be re-filtered.
From one egocentric human video to a robot policy evaluated in the twin.
The policy (c-hist-ev) is pretrained on 1,691 h of human egocentric video. Arm C starts from the base video model instead, without this step.
A single HumanEgo recording, serve_bread_002: a person picks up a croissant and puts it on a plate. It defines the task and the object trajectories.
The recording is replayed object-centrically in the Isaac twin under randomized placements and physics. The agent-tuned controller (cem8) turns hand motion into Sharpa finger commands, and only strict successes become finetuning data (arms A and C).
The finetuned policy sees the ZED image and the measured joint state. It drives the Tianji arms and Sharpa hands through the same controller, 32 executed steps per call and up to 35 s per rollout, on 30 placements it was not trained on.
Each test case starts from an egocentric HumanEgo recording of a person serving a croissant. The croissant and the plate are reconstructed as textured meshes at metric scale (SAM 3D), and the person's hand trajectory is mapped onto the robot cell in Isaac Sim, with the croissant simulated as an FEM soft body.
Replaying the human motion through the robot's controller, the GT replay, is the reference for everything else.
An agent proposes batches of controller settings (grasp, squeeze, release and gain knobs), runs every candidate on the same test cases with paired seeds, and reads back where each episode failed. Screening batches are small, so every rate is shown with its Wilson 95 % interval, and the conservative estimate (the worse of two seed replicas) as a hollow marker.
Physics parameters (friction, croissant stiffness) are not tuning knobs. The first search moved them too, which makes the simulation easier rather than the controller better, so the current objective moves only controller knobs and randomises physics within an uncertainty band.
A tuned controller is reported here only after it is confirmed against the default on a validation batch of at least 60 episodes under that objective; until then the page shows the baseline, the method and the screening batches, labelled as such.
A learned policy drives the same twin from the ZED view. Every rollout on this page was precomputed in Isaac Sim: move a knob, drag on the table map or click a dot, and the viewer jumps to the nearest recorded initial condition. Nothing is simulated in your browser.
Each rollout is an r2s.viz.v1 record written by the Isaac recorder (30 Hz rows: measured joint angles, object poses, hand poses, fingertips, 16-region tactile, phase, policy calls with predicted wrist trajectories, ZED intrinsics / extrinsics, run metadata), packed for the web into episode.json + arrays.bin. The viewer interpolates joint angles and slerps object rotations between rows and renders at the display rate. Every gallery card (and the viewer toolbar) has a download button: the replay video (an MP4 rendered frame by frame in your browser), the policy-input video, and the data as episode.json + arrays.npz (NumPy) with a README.
| Space | play / pause |
| ← → | step one row (Shift: ten) |
| 1 2 3 4 | Hero / Close / Top / ZED camera |
| c | close-up inset on / off |
| o | policy overlay (predicted wrist path, human fingertips) |
| f | fullscreen viewport |
| ? | this help |
| Esc | close compare / help |
| drag · scroll | orbit · zoom (turns the auto camera off; ↺ restores it) |