Evaluating coding agents on physical-world reconstruction.
Agents turn robot recordings into interactive simulations. We evaluate the reconstructed scene against the recording, then test it with recorded actions and a fixed robot policy.
A video could not load. Open this run to try the individual recordings.
| Rank | Model | Harness |
|---|
Inspect all 60 task, model and harness combinations, with scores and open-loop and closed-loop rollouts.
Each agent reconstructs the same recorded episode. We evaluate the final scene without human repairs.
Videos · calibration
actions · task instruction
A DROID episode supplies three timestamped camera videos, stereo calibration, seven-joint and gripper commands, and the original task instruction. The benchmark covers ten episodes recorded on the same Panda and camera rig.
Two conditions, the same task
Both conditions use the same simulator toolkit and native tools. Each agent builds its own scene, including the robot and cameras. Both conditions get the same task text. Raw adds generic coding instructions. RoboSimStudio adds its released AGENTS.md, which links its construction skill.
The agent edits the scene, runs simulation and inspects renders at provider-native medium effort. All 60 constructions ended with the model finishing. Astra Tasks 2–4 were rebuilt uncapped on a newer worker image; the earlier capped runs are superseded.
Scene · assets · physics
camera poses · scene hash
The final candidate is frozen with its assets and a scene hash. Its native physics and camera poses are preserved. Evaluation starts from this scene in fresh processes, with no repairs or score feedback to the agent.
Both runs reported the can in the bowl. The evaluator's replay agreed with only one.
Replay the original joint and gripper targets on the simulator’s native control clock. The published replay score uses the reviewed task outcome, judged against the source recording and task instruction.
Run the unchanged π0.5-DROID checkpoint for 40 seconds at 15 Hz. It executes eight actions per query, with calibrated arm gains of 320 stiffness and 100 damping.
The policy ran on 57 scenes. Three Argon Raw tasks received construction-fault zeros without a policy rollout.
We define 4 metrics to evaluate the Real2Sim quality, and create an equally weighted combined score.
Combined = (visual + semantic + open loop + closed loop) ÷ 4
1 − mean LPIPS
AlexNet LPIPS over nine time-aligned real and simulated frame pairs from the principal exterior camera. The score is clipped to [0, 1].
Whole-image comparison, including the robot and background.
Mean DINOv2 cosine similarity
The revised metric compares normalized CLS embeddings from DINOv2 ViT-g/14 across the same nine frame pairs, then clips the mean to [0, 1]. It replaces Grounding DINO + SAM mask IoU.
A whole-image similarity measure; it does not verify object identity or placement. DINOv2 scores are measured across all 60 runs.
Reviewed task outcome · 0 or 1
Dhanvi Reddy reviewed the replay videos against their source recordings and made the final outcome calls. Disputed cases were checked against saved rollout states; runs not flagged in the review retain their automated result.
The review had one reviewer who knew the model and harness. Automated geometric rules were not revised to match the review.
Policy task success · 0 or 1
A task-specific geometric predicate evaluates the fixed policy’s final state over the final half-second. Policy contract and arm gains are recorded with the measurement.
The policy has no passes and no recorded known-good-scene control, so its scores cannot distinguish model capability from policy or evaluation limitations.
| ID | Task | Instruction | Source |
|---|
Final results from 60 runs across three models, two construction conditions and ten tasks.
With RoboSimStudio guidance, Opus led visual fidelity (0.580), semantic fidelity (0.760), and open-loop replay (10/10). It also had the highest visual mean across both construction conditions and reproduced the task in 16/20 reviewed replays.
Astra and Argon showed different strengths. Astra scored higher on visual and semantic similarity on average, while Argon reproduced far more interactions: 13/20 reviewed replays versus Astra’s 3/20. Astra Raw had the highest semantic mean, 0.779, so Opus did not lead that metric under every condition. Image similarity and physical replay are distinct capabilities.
Guidance increased reviewed replay success from 11/30 with Raw prompts to 21/30 with RoboSimStudio, a gain of 33 percentage points.
The improvement was uneven. Cup nesting went from 0/3 to 2/3 successes, moving two cans from tray to counter from 0/3 to 3/3, and placing cans on opposite sides of a tray from 1/3 to 3/3. Several other tasks were unchanged, including removing cups from a tray. These examples suggest that guidance can help coordinate more involved reconstructions; difficulty tiers were not defined in advance.
None of the executed policy rollouts completed its task. A reconstruction could reproduce a recorded interaction and still fail when the real-data-trained π0.5-DROID policy chose its own actions.
That leaves a substantial end-to-end transfer problem unresolved. A policy needs compatible visual observations and physical responses, beyond a convincing render or one successful replay. Because we have not yet validated the policy on a known-good simulated scene, these failures may also reflect the policy or control interface; they do not isolate the visual or physical gap.
The newer construction-cost records separate each final conversation from discarded restarted attempts. Across the 60 final conversations, estimated API usage totals $2,639.19.
Final-conversation API cost per scene averaged $68.80 with Raw and $84.98 with RoboSimStudio for Opus, $13.33 and $15.78 for Astra, and $43.58 and $37.46 for Argon. Allocated API expenditure per reviewed replay success was $96.11 for Opus, $97.03 for Astra and $62.33 for Argon.
Per-scene figures average ten final conversations per condition. API cost per reviewed replay success divides each model’s combined final-conversation spend by its reviewed passes. These are usage-based estimates, excluding GPU costs, discarded restarted attempts, and unresolved charges.
Astra was cheapest per reconstruction, but its low replay success erased that advantage on a cost-per-success basis. Argon had the lowest API expenditure per reviewed replay success, while Opus delivered the strongest visual fidelity and replay performance.
RoboSimStudio cost about 10% more in final-conversation API usage overall, yet produced 21 reviewed successes versus 11 for Raw. Allocated API expenditure per reviewed replay success fell from about $114 to $66. Better guidance can improve the economics by increasing the share of outputs that reproduce the interaction.
Median tokens for each model and condition across the ten tasks.
Opus used more tokens on longer recordings: a median of 71k at 24 s or less and 111k at 31 s or more. Astra and Argon did not.
Compare your coding agent or construction harness on the same recorded tasks and evaluation protocol.
Evaluate a model or harnessExpand the benchmark with recorded episodes, calibration and actions, or a new robot model and fixed policy.
Contribute a task & embodiment