Add a VLA as a learned skill

Any policy XPolicyLab can serve becomes a learned skill (run_policy) with one manifest: XPolicyLab’s own policy server runs it, and run_policy speaks its protocol, handing it the observation RoboDojo’s evaluation client would. The agent’s policy is Simate ([services] policies = ["simate"] in configs/default.toml); this tutorial adds G0.5 next to it, or in its place.

1. The manifest

services/g05.toml:

kind = "policy"
summary = "G0.5 VLA, RoboDojo's own checkpoint (joint actions, both arms), through XPolicyLab's G05 adapter"
protocol = "xpolicylab"
port = 19005
# Measured on a 3090: 16.4 GB. The first request after a start takes 15-50 s.
vram_gb = 16.4
load_seconds = 120
needs_files = ["{weights}/g05/checkpoints/checkpoint", "{weights}/g05-processor/tokenizer.json",
               "{xpolicylab}/policy/G05/G05"]
notes = """The checkpoint config points at a processor path that is not shipped; OpenGalaxea/G05 on Hugging Face \
is gated, and its qwen3_5_2b_base_processor files match Qwen/Qwen3.5-2B-Base (ModelScope) byte for byte. \
The description (what the model is told) names kinds of manipulation, never tasks: it generalises G0.5's \
own scores run alone through run_policy on development layouts (configs/experiments/policy-alone-dev.toml; \
scripts/policy_card.py sorts them), 2026-10-13: 2 layouts of each task, 4 for the five tasks whose kind moved \
from the earlier description (derived from official evaluation pack 0); 2 reliable, 12 partial, 24 unreliable. \
Not measured: cover_blocks, swap_blocks and play_tic_tac_toe (every arrangement is an evaluation layout, so \
none is left for development) and imitate_sorting_sequence (RoboDojo marks both layouts unstable mid-episode)."""

[client]
url = "{url}"
description = """A visuomotor policy trained on whole RoboDojo tasks: each demonstration starts at the start \
pose and finishes the task, labelled with the task's own instruction; it moves both arms (joint actions) with \
skilled contact, many times a second. It does best in exactly that setting: from the untouched start pose, with \
the task's own instruction, left to run. It does not reason: it cannot read words, numbers or pictures to decide \
what goes where, does not remember what it saw earlier, and does not notice its own mistakes. \
Reliable (it usually finishes by itself): putting a few separate objects into a bin, case or box. \
Partial (it often does part and leaves the rest): stacking blocks in a pile or tower; folding cloth; pouring a \
liquid or small loose objects into a container; nesting bowls or cups; tidying objects onto their places; \
filling a holder with several items; putting an item into an appliance; packing several objects into a box; \
standing tubes upright in a rack or driving a screw (rarely all the way). \
Unreliable (it rarely helps): hanging items on a hook or rack; sorting objects into groups; filling a tray of \
small slots; building a shape from parts; pushing or sweeping objects to an exact pose; any task where what \
goes where depends on reading words, numbers, an equation, a picture or a demonstration; remembering what \
happened earlier (hidden, swapped or shown objects, buttons in order); catching objects on a moving conveyor; \
small precise insertions (a key, coin or plug); ordering by size; playing notes or stacking a toy in a given \
order. \
Signs it is failing: the arms stop moving while objects are not where the task needs them; the same reach is \
repeated; an object is knocked over or off the table; the jaws close fully on nothing after carrying something."""

[xpolicylab]
policy = "G05"
ckpt_name = "RoboDojo-sim-arx_x5-joint-0"
action_type = "joint"
cwd = "{xpolicylab}/policy/G05/G05"
pythonpath = ["{xpolicylab}/policy/G05/G05/src", "{xpolicylab}/policy/G05/G05", "{robodojo}", "{xpolicylab}"]

[xpolicylab.overrides]
ckpt_path = "{weights}/g05/checkpoints/checkpoint"
g05_root = "{xpolicylab}/policy/G05/G05"
action_source = "fm"
hydra_overrides = "['model.model_arch.hf_processor_path={weights}/g05-processor','model.processor.tokenizer_params.pretrained_model_name_or_path={weights}/g05-processor']"

[[weights]]
dir = "g05"
modelscope_dataset = "RoboDojo-Benchmark/RoboDojo"
root = "ckpt/RoboDojo/G05/RoboDojo-sim-arx_x5-joint-0"

[[weights]]
dir = "g05-processor"
modelscope_model = "Qwen/Qwen3.5-2B-Base"
exclude = ["*.safetensors", "*.bin", "*.pt"]

Key	What it is for
`protocol = "xpolicylab"`	no `command`: the server is XPolicyLab’s `setup_policy_server.py`
`[xpolicylab]`	the policy folder under `XPolicyLab/policy/`, the checkpoint, the action type (`joint`, or `ee` for a policy that outputs link6 poses, which `run_policy` solves by IK every step, as for OpenWAM-alpha in `services/openwam-policy.toml`), `overrides` of its `deploy.yml`, `pythonpath` and `cwd` when the policy needs them
`vram_gb`, `load_seconds`	measured: placement and swapping use them
`client.description`	what the model is told (`list_policies()`), and what it decides the strategy from: the kinds of manipulation the policy does reliably, partly or not at all, what it cannot do, and the signs that it is failing; never task names (step 3)
`client.scoped`	the policy keeps a memory per evaluation id: `run_policy` gives each episode its own (Simate)
`notes`	for people: where the description came from
`[environment]`	the environment the policy server runs in: its Python version, pinned requirements and the sources to check out (XPolicyLab’s policy code at a fixed commit); built the first time a run uses the policy, apart from every other service
`[[weights]]`	where the checkpoint comes from (ModelScope, Hugging Face); downloaded with the environment, checked by sha256
`needs_files`	what must exist before it can start

{weights}, {xpolicylab}, {robodojo}, {repo}, {python}, {port}, {url} are filled in from the deploy.

2. Install and placement

The first run that uses the policy installs it: its environment, its sources and its checkpoint. To do it ahead of time:

harness install g05

The harness places it on a GPU with room for it. To choose, give it a GPU in a deploy, or put it on demand in one swap group with Simate:

[services.g05]
gpu = 2

harness doctor g05 --deploy mine
harness up g05 --deploy mine

3. Its description

The model decides from the description alone, so it says what the policy does when handed a whole task from the start pose, measured, never what we hope. The descriptions in services/ come from development layouts: the policy alone on every task, then its per-task scores sorted into kinds.

harness run configs/experiments/policy-alone-dev.toml --set 'agent.services.policies=["g05"]' \
  --script configs/scripts/policy_alone.json --output runs/g05-alone-dev
python scripts/policy_card.py runs/g05-alone-dev

policy_alone.json hands the configured policy the whole task from the start pose and lets it run to the end of the episode, with no model calls. policy_card.py sorts the tasks into reliable (mean score at least 0.7), partial (at least 0.1) and unreliable, with the tasks behind each. Write the description from that, generalised into kinds of manipulation (“nesting bowls or cups”, not stack_bowls): the leaderboard allows no router keyed on tasks, and the model meets tasks it has no line for.

What it decides: the prompt keeps strategy B (the programs’ own tools) for picking one object out of several by what identifies it, among objects that stand still, and hands every other task to the policy. The kinds choose between A (hand over and supervise: the policy usually finishes that kind) and C (the policy first, then finish what it leaves). Say also what the policy cannot do, so that the model keeps those tasks, and the signs that it has stopped helping.

4. Use it in the agent

description = "the agent with Simate and G0.5"
include = ["dev.toml"]

[overrides.agent]
services = { policies = ["simate", "g05"] }

Programs then see both in list_policies(). start("A" or "C", kind) hands the task to the first one unless it names another (policy="g05"), and run_policy takes them by name:

print(list_policies())                     # name, description, arms of each policy
trace = run_policy("g05", max_seconds=8)   # the task's own instruction, both arms
print(trace.event.kind, get_state()["step"])

With policies = ["g05"] it replaces Simate (another identity, as any change of the agent).

run_policy(name, instruction=None, arms=ARMS, max_seconds=None, region=None, until=None, reset=False, stall_seconds=5.0, from_anywhere=False) is a bounded execution like any motion: guards, until, region, the shield and handlers apply. Every tick the policy gets RoboDojo’s own observation, and each action chunk is executed whole, as RoboDojo’s evaluation client does. A fresh run (the first, another instruction or arms, reset=True) starts only from the start pose the demonstrations start from; elsewhere it ends at once with SKILL_ERROR (“not at the start pose”; from_anywhere=True lifts that for R1). Calls with the same policy, instruction and arms continue where the last one stopped (the policy keeps its state, and the unused rest of the action chunk is kept), so slices of max_seconds equal one long run. Without max_seconds it runs until the episode ends, until holds, or the arms stop moving for stall_seconds once they have moved (a normal end: COMPLETED, detail stopped = "stalled"). The agent’s runtime.policy_progress_seconds (0, off, in the default agent) also ends a run in which the objects the last locate tracks have not moved 3 cm for that long (stopped = "no_progress").

5. Check it without a model

configs/scripts/run_policy_check.json is a scripted program that lists the policies, runs the first one for 6 s and prints the joints and the wall time. Replay it against a running worker (the experiment’s endpoint) with the new policy configured:

harness program configs/experiments/dev.toml --set 'agent.services.policies=["g05"]' \
  --script configs/scripts/run_policy_check.json --task stack_bowls

configs/scripts/run_policy_rules_check.json checks the start-pose rule the same way: a fresh run refused away from the start pose, run after home(), continued until the arms stop.