Submit to RoboDojo's leaderboard

RoboDojo’s leaderboard runs submitted policies through XPolicyLab. This tutorial takes the agent from development to a pull request. Evaluation through XPolicyLab explains the parts involved.

The leaderboard’s rules that shape a submission (protocol of 2026-10-14):

one fixed system and inference procedure for every task; an agent inside it is allowed, a router that picks policies or checkpoints by task is not;
RoboDojo scores the submission itself, with its official client: three evaluation seeds (mean ± std) on the public layouts, plus hidden verification layouts (a large gap invalidates the entry);
what to submit: the inference code in XPolicyLab’s standard (a pull request), the complete local results of bash robodojo.sh summarize, a short description of the approach and the model types it uses (here: agent, VLA, LLM/VLM), and the commitment to make code, configuration and checkpoints public within 3 days of listing;
to be listed, the approach needs something new (an architecture or method, which may be a principled integration of existing models) and a paper or technical report, or a concrete plan for one.

1. Develop under the contract

The official client hands a policy less than our worker reports: no camera calibration, grippers as their last commands, no step limit, no task name. The agent sees exactly that by default (configs/default.toml):

# The agent: what `harness run` runs unless an experiment names another agent file.
# Every layer of the harness is on: programs over analytic and learned skills (Simate),
# the reactive layer (guards and controllers every tick), handlers, and the success
# verifier, above the fixed safety shield. It sees what RoboDojo's official client hands
# a policy, nothing more (the leaderboard protocol), and it is the configuration
# submitted to the leaderboard. The ablations of the paper (configs/ablations/) include
# this file and switch parts off.
description = "Harness Policy: programs over analytic and learned skills, reactive layer, handlers, verifier"
prompt = "../prompts/robodojo"
# The model roles (configs/models/<name>.toml); an experiment's `model` or --model NAME
# picks another.
model = "deepseek-flash"
# No pointer role: the vlm role (the main model) points, as in every 1.4 evaluation run,
# where Molmo2 was not running and programs pointed in 3 of 129 episodes.
roles = ["main", "vlm", "verifier"]

[skills]
layers = { learned = true, reactive = true, handlers = true }

[services]
segmenter = "sam3"
# The learned policy: Simate. Inside the agent it scored as G0.5 did on the development
# comparison; alone it scored more over all 42 tasks (Long-Horizon 38.9 against 19.1).
policies = ["simate"]
# No world model: no 1.4 evaluation run had OpenWAM, and without it the models fit one
# 24 GB GPU (kinematics, SAM 3, Simate: 23.2 GB). `--set 'agent.services.world_models=
# ["openwam"]'` adds predict_future.
world_models = []

[runtime]
# Long enough for pointing calls inside a program to wait out a network outage.
cell_timeout_seconds = 1800
reactive_budget_ms = 20.0
default_max_seconds = 30.0
cameras = ["cam_head", "cam_left_wrist", "cam_right_wrist"]
verifier = true
# The first motion is start(strategy, kind): hand the task to the learned policy (A, C)
# or not (B), decided from its description before anything moves (Z2's first call).
strategy_first = true
# The handed-over task runs uninterrupted: no progress watch (policy_progress_seconds
# 0). Stopping the policy early bought nothing (the model's own tools after it added no
# score), and measuring the scene while it ran may harm it: pour_liquid_into_cup,
# development layouts 0-1, policy alone 4 of 4 (2 scripted now, 2 on 13 October), with
# the watch measuring 1 of 4 (2 scripted, 2 of agent 1.4.0). The stall stop still ends
# a run whose arms moved and then stopped.
policy_progress_seconds = 0.0
# After the handed-over task's first run the model has this many calls to finish (its own tools, the policy
# again, or finish); then the policy runs on to the end of the episode. Finishing with
# the tools took about half of the development run's spend for no score over the policy
# alone.
post_policy_calls = 4
# The XPolicyLab observation contract (core/contract.py): instruction, camera colors,
# measured arm joints, link6 poses, the last gripper commands; camera calibration from
# the embodiment; no depth, no step limit, no task name, so no task descriptions
# (configs/ablations/with-task-description.toml adds them, Tab. 4's second row).
# `--set agent.runtime.contract=native` shows a development run everything the
# simulator worker measures.
contract = "xpolicylab"
# Programs run isolated where the machine allows user namespaces (an evaluation host
# need not).
isolation = "auto"

[budget]
# 30 model calls per episode (Zhang et al. 2026b ran 100). A handed-over task needs a
# few; the model's own tools where they fail used the rest of 100: agent 1.4.0 on
# evaluation layouts, pick_from_conveyor_by_image 98 calls ($0.38) for 0.
max_model_calls = 30

harness run configs/experiments/dev.toml --deploy mine --output runs/dev

Results on development layouts are what the exported policy should reach.

2. Choose the models and export

The export freezes the agent with its model. Pick the model of the submission (--model NAME, a file of configs/models/; the agent’s own is DeepSeek V4.1 Flash through OpenRouter), commit and push, then:

harness export xpolicylab --name HarnessPolicy \
  --out <RoboDojo>/XPolicyLab/policy/HarnessPolicy --deploy mine

The agent is the submission as it is: no pointer role (the VLM, the main model itself, points) and no world model, so the package needs the kinematics service, SAM 3 and Simate, 23.2 GB, which one free 24 GB GPU holds resident. The evaluator needs only OPENROUTER_API_KEY besides the checkpoints. harness lock prints the identity the package will record before you export (agent 1.6.0 with DeepSeek V4.1 Flash: 3786ebbcd0e640ae); development runs of the agent have the same identity, so their results are the package’s. A --set or --model on the export changes it, as on any run. (Agent 1.4.2 was exported with --set 'roles=["main", "vlm", "verifier"]' --set 'services.world_models=[]', which 1.5.0 made the default.)

The export refuses an agent that does not run under the contract or reads task descriptions, one that does not resolve strictly, and uncommitted code (--allow-dirty overrides the last, for trials). --vendor copies the files the agent was resolved from into the folder instead of letting install.sh clone the commit: use it while the repository is private, since the evaluators’ install.sh cannot clone it.

3. Check

harness export xpolicylab --name HarnessPolicy \
  --out <RoboDojo>/XPolicyLab/policy/HarnessPolicy --deploy mine --check --gpu 5

XPolicyLab’s PR checks: bash -n on every script, py_compile, no image decoding in model.py or deploy.py, and ten episodes of XPolicyLab’s debug environment with scripted model replies (configs/scripts/xpolicylab_check.json). The package places its services on the GPUs it is given by itself; one 24 GB GPU is enough (the models load on demand).

4. Run the official client

With <RoboDojo>/XPolicyLab/policy/HarnessPolicy in place, RoboDojo’s own client runs it exactly as the evaluators will:

cd <RoboDojo>/XPolicyLab/policy/HarnessPolicy
export TMPDIR=/data/me/tmp            # Isaac Lab writes under $TMPDIR/isaaclab
EVAL_NUM=1 bash eval.sh RoboDojo general_pickup HarnessPolicy arx_x5 joint 0 5 5 harness-policy harness-policy

Arguments: bench, task, checkpoint name (not read), robot, action type, seed, policy GPUs, simulator GPU, policy environment, simulator environment. EVAL_NUM caps the episodes. The client reads layouts from Assets/Eval_Layout/RoboDojo/arx_x5/<seed>/<task>_<n>.json; to check the wiring without touching evaluation layouts, serve a development layout as an unused seed:

mkdir -p <RoboDojo>/Assets/Eval_Layout/RoboDojo/arx_x5/9000
ln -s <dev_layout_dir>/general_pickup_3.json <RoboDojo>/Assets/Eval_Layout/RoboDojo/arx_x5/9000/general_pickup_0.json
EVAL_NUM=1 bash eval.sh RoboDojo general_pickup HarnessPolicy arx_x5 joint 9000 5 5 harness-policy harness-policy
rm -r <RoboDojo>/Assets/Eval_Layout/RoboDojo/arx_x5/9000

Each policy-server instance writes its traces to results/<task>-<time>-<pid>/ in the policy folder (one directory per episode, as harness run writes them, but without the outcome: the host scores); RoboDojo writes its videos and scores under eval_result/.

A check of agent 1.4.1’s package this way (one development layout per task served as seed 9000): stack_bowls, general_pickup and make_toast succeeded, solve_equation did not; no request ran into the client’s 120 s limit.

For the application, run one complete evaluation locally: every task and _random variant of a pack at the official episode counts (EVAL_NUM unset), then summarize it with RoboDojo’s bash robodojo.sh summarize. RoboDojo runs the three seeds itself. A policy server needs its services’ GPU memory back before the next eval.sh starts on the same GPU: wait until the previous instance’s services have exited.

5. Submit

Open a pull request with policy/HarnessPolicy/ against XPolicyLab. The folder pins the commit (agent.json, install.sh); its README follows XPolicyLab’s adapter template.
Apply at robodojo-benchmark.com/eval: the pull request’s URL and commit, the policy and checkpoint, the benchmark, tasks, robot and action space (arx_x5, joint), the environment variables and downloads the package needs (OPENROUTER_API_KEY: explain that it is a hosted model’s key), the local results and the description of the approach.

The evaluators run install.sh, download_checkpoints.sh, set OPENROUTER_API_KEY, then eval.sh. At start the policy resolves agent.json again and refuses to run unless it gets the identity that was exported.