Suites and reports

harness_policy.runtime.suite

Run an experiment over many tasks and seeds: harness run EXPERIMENT.toml.

One RoboDojo (or RoboTwin) worker serves one task, so each slot (a GPU of the deploy’s [sim] x workers_per_gpu) starts a worker for its next task, runs that task’s seeds, then stops the worker. Finished episodes (summary.json present) are skipped, so a rerun resumes. Results go to OUTPUT/<task>/seed-<n>/; summarize with harness report OUTPUT. Run inside the harness-policy environment.

log

def log(output, message)

Print message with the time and append it to OUTPUT/suite.log (one slot at a time).

robotwin_tasks

def robotwin_tasks()

The 50 RoboTwin 2.0 tasks, from the official per-task step limits.

expand_tasks

def expand_tasks(spec, benchmark='robodojo')

The task names of a comma-separated spec, in order and without repeats. On RoboDojo an item is a task, a dimension name, “moving” (the moving-target tasks) or “all”; on RoboTwin a task or “all” (its 50 tasks).

expand_seeds

def expand_seeds(spec)

The seeds of a comma-separated spec of seeds and inclusive ranges (“0-4,7”).

worker_options

def worker_options(args, task, directory)

Write the worker’s options for task to <directory>/worker-<task>.toml and return the path: RoboTwin’s task config, split and seed group, or RoboDojo’s video, ground truth, shield and layout settings.

start_worker

def start_worker(args, task, gpu, port, output)

Start a simulator worker for task on gpu, serving on port in its own process group (log: workers/worker-<task>.log), and return the process once it answers. Raises RuntimeError when it exits, or stops it and raises when it is not ready in time.

stop_worker

def stop_worker(process)

Stop a worker’s process group if it still runs: SIGINT, then SIGKILL after 30 s.

done

def done(directory, rerun_errors)

An episode counts as done once it has a summary; with rerun_errors, one that ended in an error (provider outage, dead worker) is run again.

replacements

def replacements(task_dir, name='unstable')

<task>/unstable.json: {unstable seed: the layout run in its place}, so a resumed suite runs the same replacement again instead of retrying the unstable layout. <task>/displaced.json: {requested seed: the layout run for it}, for a requested layout an earlier suite in this output had already run in place of an unstable one (that run keeps counting for its owner).

write_replacements

def write_replacements(task_dir, name, mapping)

Write <task>/<name>.json atomically: a kill mid-write must not leave broken JSON for the resume.

pack_size

def pack_size(args, task)

Number of layouts of task in the eval pack, or None when the checkout is not readable here.

run_slot

def run_slot(args, slot, gpu, port, tasks, output, run)

One slot’s loop: take tasks from the queue until it is empty, and for each start a worker, run the seeds not done yet, and stop it. On RoboDojo eval layouts an unstable layout is replaced by the pack’s next unused one; another worker error restarts the worker, and once the suite’s cost cap is reached the remaining episodes are skipped.

experiment_for

def experiment_for(run, task, seeds)

The driver’s configuration of one task of the experiment.

suite_arguments

def suite_arguments(run, *, gpus=None, workers_per_gpu=None, base_port=None, rerun_errors=False, script=None, max_total_cost_usd=None)

The suite’s settings: the run’s, the deploy’s [sim] section, then the command line.

preflight

def preflight(run, log=print)

Check that the services the experiment’s agent uses are up before any simulator boots: resident ones must answer (harness up), on-demand ones need the deploy’s broker (started here when it does not run).

run_suite

def run_suite(run, output, **options)

Run every task and seed of run into output (see suite_arguments for options).

harness_policy.runtime.report

Aggregate program-harness experiments into RoboDojo Score / SR tables.

RoboDojo scores an episode 1 on success and process_score / 100 otherwise. Score and SR are means over episodes (x100) per task; a dimension is the mean of its tasks and the overall number is the unweighted mean of the five dimensions. Generalization tasks pool their _random variant with the base task. Errors stay in the denominator as zero-score episodes, except episodes the simulator itself dropped (RoboDojo “unstable after reset/step”: neither a success nor a failure of the policy, which RoboDojo’s own evaluator leaves out); those are counted as dropped.

base_task

def base_task(task: str) -> str

The task without its _random suffix, so a variant pools with its base task.

load_episodes

def load_episodes(directories) -> list[dict]

The episode summaries under directories: each summary.json with a task and a seed in a directory named seed-<n> (superseded runs kept beside it are left out).

dropped

def dropped(episode: dict) -> bool

An episode the simulator dropped as unstable (see the module docstring).

episode_score

def episode_score(episode: dict)

(score 0-1, success) of one episode as RoboDojo counts it, or None for an episode the simulator dropped. Errors count 0 and never as a success.

aggregate

def aggregate(episodes: list[dict]) -> dict

Score and SR (x100) per task, per dimension (and for the moving-target tasks) and overall, with errors, dropped episodes, and per-episode model calls, cost, decisions and events. A dimension averages its tasks that have episodes; overall is None until every task of every dimension has one.

markdown

def markdown(result: dict) -> str

An aggregate result as Markdown: the dimension table with the average, the episode count with per-episode calls, cost and events, and the per-task table.