Quick tour

Two real episodes, from the first model request to the outcome: agent 1.4.1 (1.4.2 changed only who takes the conveyor tasks; 1.5.0 made the setting of these runs, no Molmo2 and no world model, the default) with DeepSeek V4.1 Flash on RoboDojo’s evaluation pack 0, under the XPolicyLab contract (commanded grippers, no task name, no step limit reported). In the first the model hands the task to the learned policy, Simate; in the second its own programs do the task. The first kind is the common one: in that run (42 tasks, three layouts each) 113 of 125 episodes were handed over, at 2.3 model calls an episode; the 12 the programs kept took 16.

One episode, from reset to outcome. Either the task is handed to the learned policy, or the program's own skills act; either way every command passes the shield.

Episode 1: a task the policy learned

“Stack the three bowls together.” (stack_bowls, layout 0)

Program 1: look

The driver sends the system prompt (the RoboDojo prompt pack, assembled for the agent’s skills), then one user message: the instruction, the robot state and the three camera images. The decision reason is start. The model answered in 28 s:

# PLAN: three white bowls to stack. Check policy list and locate bowls first.
print(list_policies())
obs = get_observation()
for o in locate("white bowl"):
    print(o)

Looking runs before start; nothing that moves the arms does. list_policies() returned Simate’s description (client.description in services/simate.toml), what it does from the start pose, by kind of manipulation, never by task name. An excerpt:

It cannot pick one object out of several by reading what identifies it: a name, a number or label
written on it, or a picture shown as the target. [...] Often does part and leaves the rest: packing
several objects into a box; stacking blocks in a pile or tower; [...] nesting bowls or cups; [...]

locate measured every bowl with SAM 3 in every camera. The loop printed the keys of the dictionary locate returns (objects, text) rather than the objects, but after every program the driver measures the located objects again and sends the scene report:

o1 'white bowl' at (-0.275, -0.066, 0.805) (seen in cam_head and cam_left_wrist; SAM 3 match 0.98),
   12 x 11 cm, long side at 83 deg, a container; reached straight down by the left arm; too wide for
   the open jaws: pick takes it by an edge.
o2 'white bowl' at (0.044, -0.127, 0.785) (seen in cam_head only, so assumed 0.02 m above the table; ...
o3 'white bowl' at (0.248, -0.039, 0.804) (seen in cam_head and cam_right_wrist; SAM 3 match 0.98), ...

The next message carried the output, the scene report, a close-up of every object (to check that each is what was named) and the camera images with the objects outlined.

Program 2: hand it over

# PLAN: three white bowls (o1 left, o2 center, o3 right) — task is nesting/stacking bowls.
# Policy lists "nesting bowls or cups" under "Often does part and leaves the rest" -> strategy C.
start("C", "stacking/nesting three bowls together")

start("C", ...) records the choice and runs the policy at once, as RoboDojo’s own evaluation client would: from the untouched start pose, with the task’s own instruction and both arms, each action chunk executed whole, a fresh observation every tick. (A: hand over and supervise, for kinds the description says it usually finishes; C: the policy first, then finish what it leaves; B: the programs’ own tools.) Simate stacked the bowls, and at step 230 RoboDojo’s success check passed and ended the episode inside the program, 173 s after it started, so no model call followed:

{"reward": 1.0, "success": true, "terminated": true, "truncated": false, "process_score": 100.0, "steps": 230}

{"status": "episode_ended", "success": true, "steps": 230, "model_calls": 2, "verifier_calls": 0,
 "cells": 2, "decisions": {"start": 1, "program_end": 1}, "input_tokens": 19874, "output_tokens": 428,
 "cost_usd": 0.0025,
 "strategy": {"strategy": "C", "kind": "stacking/nesting three bowls together", "policy": "simate",
              "step": 0, "policy_ran": true}}

Had the policy stopped with the task undone (its arms still for 5 s), the program would have ended there and the model would have seen the scene measured again, with 4 calls left (post_policy_calls) for what is clearly left; after them the policy runs on to the end of the episode.

Episode 2: a task the programs keep

“Pick up the blue race toy car by 10 cm.” (general_pickup, layout 4)

Program 1: choose, then look

# PLAN: Task = pick up the blue race toy car and lift it 10 cm. It is named in the
# instruction -> strategy B, my own tools. Step 1: start(B), locate the blue car.
start("B", "pick up the blue race toy car by 10 cm")
res = locate("blue toy car; blue race car")
print(res["text"])
for o in res["objects"]:
    print(o["id"], o["name"], o["position"], "reach:", o["reach"], "jaws:", o["jaws"], "size:", o["length"], o["width"])

The table holds other objects too; which one to take is read from the instruction, so the prompt’s rule sends the task to B. start("B", ...) records the choice and moves nothing. Two names SAM 3 matched to the same pixels are one object:

o1 'blue toy car' (also what 'blue race car' matched: the same pixels) at (0.255, -0.170, 0.790)
   (seen in cam_head and cam_right_wrist; SAM 3 match 0.91), 10 x 6 cm, long side at -13 deg;
   reached straight down by the right arm; the jaws close across its width.

Program 2: an error

r = pick("right_arm", "blue toy car", lift=0.10)

ValueError: No object 'blue toy car' in the scene; locate first (ids: o1)

The tools take the ids of the scene report. The program raised, the kernel kept the traceback to the program’s own frames, and the model was called with reason error.

Program 3: one contact

# PLAN: o1 blue car at (0.255,-0.170,0.790). Use id "o1". Right-arm pick, lift 0.10, then finish holding.
r = pick("right_arm", "o1", lift=0.10)
print(r)
print("holding:", holding("right_arm"))

pick is one contact, and the program ends after it so the model can look. Each skill call inside it is a bounded execution that ends with an event:

Execution	Ticks	Event
move_eef (6 cm above the grasp, jaws turned across the car)	23	COMPLETED
move_eef x4 (down: 3 cm above the grasp at speed, then 2, 1, 0)	7, 9, 7, 9	COMPLETED
close_gripper, hold (squeeze 0.24 s)	10, 6	COMPLETED
move_eef (lift 10 cm)	11	COMPLETED

{'ok': True, 'arm': 'right_arm', 'object': 'o1', 'grasp': 'across its width',
 'grasp_point': [0.266, -0.1787, 0.7777], 'yaw_deg': 77.1, 'pitch_deg': 0.0, 'lift': 0.1,
 'check': "'blue toy car' rose with the jaws (measured with SAM 3): held",
 'note': 're-measured from above, the object was +11, -9 mm off; the grasp followed'}
holding: True

From above the grasp the wrist camera measured the car again and the grasp moved by the difference. The contract gives no measured gripper opening, so the pick check measures whether the object rose with the jaws; only when it cannot does the vision model look at the wrist camera, as holding() does. The scene report that followed showed the car (now o2) held by the right arm 8 cm higher, with the grip:

Grip, right_arm holds o2 'blue race car': measured now in the air, the held object's centre is
(-0.007, +0.006) from the grasp point (on the table before the pick it was (+0.000, +0.000));
release applies this offset. Its lowest point hangs about 0.013 m below the grasp point.

Program 4: finish

# PLAN: Task = pick up the blue race toy car by 10 cm. Done: right_arm holds o2 'blue race car'
# lifted ~0.10 m (grasp point 0.778 -> now 0.871). Task is to lift it, so keep holding and finish.
print("holding right:", holding("right_arm"))
finish("Picked up the blue race toy car with the right arm and lifted it 10 cm (held in air).")

finish ended the program. The driver ran the wrap-up, which most RoboDojo tasks need: each gripper that holds nothing opens (the right one held the car and stayed closed), then both arms return to their start pose. RoboDojo ended the episode on success during it, at step 95. The driver read the outcome with a token the model never sees:

{"reward": 1.0, "success": true, "terminated": true, "truncated": false, "process_score": 0.0, "steps": 95}

{"status": "finished", "success": true, "steps": 95, "model_calls": 4, "verifier_calls": 0,
 "cells": 4, "decisions": {"start": 1, "program_end": 2, "error": 1}, "input_tokens": 43673,
 "output_tokens": 1601, "cost_usd": 0.0029, "vision_calls": 2,
 "strategy": {"strategy": "B", "kind": "pick up the blue race toy car by 10 cm", "policy": null, "step": 0}}

Four model calls, four short programs: look, a wrong argument, one contact, finish. vision_calls counts the vision requests made inside the programs (the two holding() looks). Programs and decision points explains the design; The episode runtime explains each part.