Run the agent on RoboDojo

This tutorial runs the agent (programs, learned skills, reactive layer, handlers, verifier) on RoboDojo: one development episode, a development subset, and the evaluation protocol. It assumes the services of your deploy are up (Quickstart).

Layouts: development first

RoboDojo’s official evaluation uses fixed layout packs (layout_pack_seed 0, 1, 2). Prompt work, skill work and self-improvement use development layouts instead: layouts = "procedural" replays <dev_layout_dir>/<task>_<seed>.json, generated by benches/robodojo/devlayouts.py with RoboDojo’s own generator and checked not to repeat any evaluation layout. Development seeds are unlimited; the same (task, seed) always gives the same layout.

Three tasks have none: cover_blocks, swap_blocks and play_tic_tac_toe place their objects at a few fixed spots, and the evaluation packs already hold every arrangement, so the generator gives up (“repeats evaluation layout”) and the worker does not start. They cannot be developed on without touching evaluation layouts. On imitate_sorting_sequence both development layouts so far are marked unstable by RoboDojo partway through the episode (about step 460), so those episodes are not scored.

One episode

harness run configs/experiments/dev.toml --deploy mine --output runs/dev

# Development: the agent with its default model on one task and development layouts
# (never evaluation layouts). `harness run configs/experiments/dev.toml
# --deploy <machine>` starts the simulator workers itself; `harness program` uses the
# worker at `endpoint`.
description = "the agent, one task, development layouts"
bench = "robodojo"
tasks = ["general_pickup"]
seeds = [0]
layouts = "procedural"
endpoint = "http://127.0.0.1:8771"

[budget]
max_wall_seconds = 5400
# Cost guard for development checks; the paper runs set no cost cap.
max_cost_usd = 4.0

The agent’s default model (configs/models/deepseek-flash.toml) is DeepSeek V4.1 Flash at medium reasoning effort: about $0.01 an episode (agent 1.4.1 on evaluation pack 0: $0.010 and 3.6 model calls on average; $0.003 and 2.3 calls when the task is handed to the learned policy, $0.08 and 16 calls when the programs keep it). --model grok-low (grok-4.7 at low effort) or another file of configs/models/ replaces it; the paper’s runs name theirs in the experiment.

A development subset

# The 8-task development subset (both conveyor tasks included) that ablations, model
# comparisons and self-improvement share (App. A); 10 episodes each on development
# layouts. An ablation runs it with --agent configs/ablations/<name>.toml.
description = "the agent on the 8-task development subset"
include = ["dev.toml"]
tasks = ["general_pickup", "stack_blocks", "stack_bowls", "classify_objects", "deposit_coin", "insert_tubes",
         "match_and_pick_from_conveyor", "pick_from_conveyor_by_image"]
seeds = "0-9"

[budget]
max_total_cost_usd = 20.0

harness run configs/experiments/subset8.toml --deploy mine --output runs/subset8 --gpus 5 --workers-per-gpu 2

harness run starts one simulator worker per slot (GPUs × workers per GPU, 6-9 GB each), each on its own port from base_port, and gives the slots tasks from a queue; a worker serves one task, then stops. It:

resumes: an episode with a summary.json is skipped; --rerun-errors runs again only the ones that ended in an error (a provider or worker failure);
caps spending: --max-total-cost-usd (or budget.max_total_cost_usd) starts no new episode once the suite has spent that much; a running episode finishes;
replaces unstable layouts on evaluation layouts: an episode RoboDojo marks unstable is moved aside (seed-<n>.unstable-<time>) and the next free layout runs.

The output is one directory per task and seed:

runs/subset8/
├── suite.json   suite.log   workers/   tmp/
└── stack_bowls/
    ├── seed-0/  manifest.json  events.jsonl  summary.json  kernel.log  videos/
    └── seed-1/  ...

Scores

harness report runs/subset8

harness report follows RoboDojo’s scoring: an episode scores its reward (1 on success, else the process score / 100); an error scores 0; an episode the simulator dropped as unstable is not scored. Tasks pool their _random variants; the five dimensions (Generalization, Memory, Precision, Long-Horizon, Open) average their tasks, and the overall score averages the dimensions, given only when every task has a score. It also reports model calls, cost and decisions per episode; --json prints everything. Each episode’s summary.json records the strategy the model chose (strategy: A, B or C, the kind it matched, the policy).

The evaluation protocol

# The agent on RoboDojo's official evaluation layouts (pack 0), all tasks, two layouts
# each (raise seeds to "0-49" for the full protocol). Evaluation resolves strictly: a
# skill or service the agent names that cannot run on this machine stops the run.
description = "the agent on official evaluation layouts"
bench = "robodojo"
tasks = "all"
seeds = "0-1"
layouts = "eval"
layout_pack_seed = 0

[budget]
max_wall_seconds = 5400

harness run configs/experiments/eval.toml --deploy mine --seeds 0-49 --max-total-cost-usd 200 \
  --rerun-errors --output runs/eval-pack0

Evaluation layouts resolve strictly: if the machine does not run a service the agent names, the run stops instead of running a smaller configuration (whose identity would differ). Every episode’s manifest.json records the identity, so the results of two machines can be compared, or not.

This runs the agent through our own simulator worker. The leaderboard runs the exported policy through RoboDojo’s own client instead, every task and _random variant at the official episode counts: Submit to RoboDojo’s leaderboard. Compare an agent with the learned policy it wraps on the same layouts: run the policy alone (configs/experiments/policy-alone-dev.toml with configs/scripts/policy_alone.json, or the same script on evaluation layouts) next to the agent, and pair the tasks.

Read a run: what is in an episode’s trace.
Configure runs: --model, --set, tasks, seeds, budgets.
Check a change without a model: replay a recorded episode.