Evaluation by others

harness_policy.connectors.xpolicylab

The harness as an XPolicyLab policy: evaluated by RoboDojo’s (or RoboTwin’s, or a real robot’s) own client through XPolicyLab’s policy-server protocol.

XPolicyLab drives: its client calls reset(), then update_obs(obs) and get_action() -> a chunk of actions, executes them, and sends the observation after each. The harness drives too: the model’s programs ask for one control tick at a time. The bridge here meets the two halves:

• To the session and the driver it is a robot worker (the Robot API: profile, state, tick, observe, policy_obs, fk, ik, plan, outcome) on a local port, so kernels, skills, isolation and traces work as with a simulator worker.
• To XPolicyLab it is the Model: update_obs() delivers the observation a waiting tick needs; get_action() hands out the action a tick asked for, one per call (the reactive layer sees every observation).
• While the model thinks or a model loads, get_action() returns an empty chunk after poll_seconds (under the client’s 120 s request limit). XPolicyLab’s standard episode loop then takes no action and asks again: the simulator does not advance, as with a simulator worker.
• After the program finishes (and the arms are home), get_action() holds the arms until the client ends the episode; that the episode goes on is not read as an outcome.

What the policy sees is the XPolicyLab contract (core/contract.py): the observation the client sends, the gripper values it reports (the last commands), camera calibration and kinematics from the embodiment (a kinematics service: RoboDojo’s planner without the simulator); neither the task nor its step limit. The shield checks every action before it leaves.

Bridge

class Bridge
    __init__(embodiment: dict, *, kinematics_url: str | None, poll_seconds: float=90.0, shield: dict | None=None, obs_timeout: float=600.0)

The rendezvous between XPolicyLab’s calls and the session’s ticks.

host_reset

def host_reset()

The client started a new episode: the session’s episode ends, the observation and any pending action are dropped, and the shield forgets its last command.

host_update_obs

def host_update_obs(obs)

The client’s latest observation (an XPolicyLab obs dict); it wakes a tick, or the session’s reset, waiting for it.

host_get_action

def host_get_action()

One action, or [] when none is ready within poll_seconds.

tools

def tools()

No tool schemas: the session uses the Robot API methods through call.

reset

def reset(task, seed)

The episode the host started: waits for its first observation.

call

def call(name, arguments)

Run Robot API method name (one of methods) with arguments for the session.

program_over

def program_over(episode=None)

The driver of host episode episode (default: the current one) finished: hold the arms until the host ends the episode. A driver that outlived its episode changes nothing.

close

def close()

Close the bridge; calls waiting for an observation or an action return.

harness_policy.connectors.package

What an exported XPolicyLab policy runs: policy/<Name>/model.py’s Model delegates here.

The package folder (written by harness export xpolicylab) holds:

agent.json       the frozen agent: what it was resolved from (agent and model
                 trees), the embodiment, the services it uses, the lock and its
                 identity, the code fingerprint
deploy.toml      where its weights are (checkpoints/, download_checkpoints.sh), and
                 optionally how a service runs here ([services.<name>] as in a
                 deploy file: an external URL, another python)
the-harness/     this repository at the exported commit (install.sh)

At start the agent is resolved again, strictly, against a deploy of the GPUs the policy server sees, and must have the identity it was exported with: whatever GPUs the evaluator has, the policy that runs is the one that was submitted. Every policy server is an instance of its own (its own ports, state and traces), so evaluators may run several tasks at once on one machine. Services that do not all fit the free memory of the visible GPUs run on demand in one swap group, one model loaded at a time: policies and world models first, then every model but the kinematics (the simulator waits while a model loads; only wall time changes). Services the instance started stop with it, also when it is killed.

relative_tree

def relative_tree(value, root: Path)

Absolute paths inside the repository written as “@HARNESS@/<relative>” (agent.json is the same on every machine).

relocate

def relocate(value, root: Path)

The inverse of relative_tree for a checkout at root.

visible_gpus

def visible_gpus() -> list[int]

GPUs of this process (CUDA_VISIBLE_DEVICES, as XPolicyLab’s setup script sets it from policy_gpu_id), else every GPU nvidia-smi lists.

free_port

def free_port() -> int

A TCP port on 127.0.0.1 that is free now (the system picks it).

package_deploy

def package_deploy(package: Path, services: list[str], instance: Path, log=print, gpu_rows=None)

The deploy of this instance, written to instance/deploy.toml: the services the agent uses, on the visible GPUs (by their free memory now), on ports of their own; all resident when they fit, else on demand in one swap group by SWAP_LEVELS. The package’s deploy.toml gives the weights and per-service settings (a service it sets the mode of keeps it).

reap

def reap(parent: int, deploy_path: str)

Stop an instance’s services once its policy server is gone (also when it was killed without a chance to stop them).

HarnessModel

class HarnessModel
    __init__(model_cfg: dict, package_dir, log=print)

The XPolicyLab Model of an exported agent.

reset

def reset()

A new episode: the last one’s program ends (its ticks report the episode over); this one’s starts with the first observation. Returns at once.

update_obs

def update_obs(obs)

Hand the client’s observation to the bridge; the first one after a reset starts the episode’s driver in a thread of its own.

get_action

def get_action()

A chunk of at most one action: the one a tick asked for, a hold after the program ended, or [] when none is ready within poll_seconds (Bridge.host_get_action).

close

def close()

Shut the instance down once: close the bridge, let running episodes write their traces (up to 60 s each), then stop the worker server, the instance’s services and the reaper.

harness_policy.export.xpolicylab

Export the agent as an XPolicyLab policy folder (harness export xpolicylab).

harness export xpolicylab --name HarnessPolicy         --out <XPolicyLab>/policy/HarnessPolicy [--vendor] [--check --gpu 5 --env harness-policy]

The folder follows XPolicyLab’s adapter standard (CONTRIBUTING.md): model.py (a thin Model that delegates to harness_policy.connectors.package), deploy.py and the eval.sh / setup_eval_policy_server.sh / setup_eval_env_client.sh trio copied unchanged from demo_policy, deploy.yml with the standard keys, install.sh, a checkpoint download script, and a README in their template. It adds the frozen agent (agent.json: what the agent was resolved from, with repository paths relative to the checkout; the lock and its identity; the code fingerprint and commit) and the deploy template; the policy refuses to run when the machine it lands on resolves a different identity. Evaluation only: no data processing or training (declared in the README).

resolve_source

def resolve_source(source=None, *, deploy=None, model=None, sets=())

(agent, bench) of an experiment file or an agent file (default configs/default.toml), resolved strictly.

check_embodiment

def check_embodiment(embodiment: str) -> dict

The embodiment a policy folder is exported for, refused unless calibrated on its rig (a profile without the key, such as RoboDojo’s simulated arx_x5, counts as calibrated) and served by a kinematics service of its own (its kinematics_service; arx_x5’s is services/kinematics.toml).

export

def export(source=None, *, name: str, out, deploy=None, model=None, sets=(), vendor: bool=False, xpolicylab=None, contributor: str='Harness Policy team', allow_dirty: bool=False, embodiment: str='arx_x5') -> dict

Write the policy folder for the agent of source (an experiment or an agent file; default configs/default.toml), for embodiment (embodiments/<name>; RoboDojo’s ARX X5 by default, another robot once its profile is calibrated).

check

def check(folder, *, gpu: str, env: str, eval_env: str | None=None, task: str='stack_bowls', script=None, timeout: float=1800.0, log=print) -> dict

XPolicyLab’s PR checks on the folder: bash -n, py_compile, the image greps, and the debug closed loop (EVAL_ENV_TYPE=debug) with scripted model replies.