Configuration

harness_policy.config

Configuration: TOML files that include others, with paths relative to the file that names them, and --set key.path=value overrides.

Layers, later ones winning (configs/README.md):

code defaults            the values the code uses when a file names none
default.toml             the agent: prompt, skills, services, runtime, budget, and
                         its default model (ablations/<name>.toml include it)
models/<name>.toml       model roles: main, vlm, pointer, verifier (an API model or a
                         service), plus the runtime settings a model needs
experiments/<name>.toml  the agent with a model on a benchmark: tasks, seeds,
                         layouts, budgets, worker options, overrides
--model, --set           the command line

and configs/deploy/<machine>.toml says where services run (services/deploy.py). Keys live in environment variables, never in these files.

read_toml

def read_toml(path: Path) -> dict

One TOML file as a tree, without following its includes.

deep_merge

def deep_merge(base: dict, override: dict) -> dict

override on top of base: tables merge key by key, anything else replaces.

load_layered

def load_layered(path, _seen=()) -> dict

A TOML file with its include = [...] files merged under it (in order; the file’s own keys win), path keys made absolute.

parse_set

def parse_set(text: str) -> tuple[list[str], object]

a.b.c=value as ([“a”, “b”, “c”], value); the value is read as TOML (a number, true, “text”, [1, 2], {k = 1}) and as a plain string when it is not valid TOML.

apply_sets

def apply_sets(tree: dict, sets) -> dict

A copy of tree with --set items written into it, in order. An item is a key.path=value string or a parsed (keys, value) pair; missing tables on the path are created, and a path through a value that is not a table is an error.

route_openrouter

def route_openrouter(tree, environ=os.environ)

Point every OpenRouter model in a harness at another base URL, when asked.

Applies to each model config with backend = "openrouter" and no base_url (so the openrouter.ai default) or an openrouter.ai base_url:

• OPENROUTER_BASE_URL (e.g. https://my-proxy.example/api/v1) replaces the URL; the key is still read from the model’s api_key_env (OPENROUTER_API_KEY).
• OPENROUTER_RELAY_URL replaces the URL with scripts/openrouter_relay.py and reads the key from OPENROUTER_RELAY_TOKEN.

Setting both is an error. Models elsewhere (self-hosted servers, other providers) are left alone. The rewritten configs go into the run snapshot, so a run records the URL it called.

harness_policy.config.agent

The agent configuration: what is evaluated, resolved for one machine.

There is one agent, configs/default.toml. It names the prompt pack, the skills (by interface layers or by name), the services skills use (segmenter, policies, world models), runtime settings, the budget, and its default model roles. An ablation (configs/ablations/<name>.toml) includes it and changes what the paper’s ablations change; an experiment may override any key.

Model roles come from a model file, configs/models/<name>.toml (the agent’s model, an experiment’s model, or --model NAME):

[main]                          the model that writes programs
backend = "openrouter"
model = "x-ai/grok-4.7"
[vlm]                           questions about images (and pointing without a pointer)
service = "qwen3-vl-8b"         a role may be a service of the deploy ...
timeout_seconds = 300           ... with its own settings on top
[pointer]
service = "molmo2"
[verifier]                      overrides on top of main (the success verifier)
[runtime]                       settings the model needs (context window, ...)

Resolving the agent against a deploy gives the driver’s main model, runtime and prompt, and a lock: everything that decides what a run computes and nothing about where it runs (URLs, GPUs, ports). The lock records the agent’s VERSION, and its hash is the configuration’s identity; a leaderboard entry, a paper table and an exported XPolicyLab policy each name one.

canonical

def canonical(value) -> str

value as compact JSON with sorted keys (other objects as their str), so equal trees give the same text.

sha256

def sha256(value) -> str

The hex sha256 of value’s canonical JSON.

code_fingerprint

def code_fingerprint(roots=CODE_ROOTS) -> dict

sha256 of every source file of the agent (Python, prompts, manifests), and of them all.

ResolvedAgent

class ResolvedAgent

The agent resolved against one deploy: the driver’s main model, verifier, runtime and prompt, the episode’s model-call budget, and the lock with its identity (the first 16 hex digits of the lock’s sha256). base_prompt is the prompt without the self-improvement library’s additions.

model_names

def model_names() -> list[str]

The model files of configs/models/, by name.

model_path

def model_path(name) -> Path

The file of a model: a name of configs/models/, or a path to a .toml file.

load_model

def load_model(source) -> dict

Model roles: a model file’s name or path (with its includes), a tree as it is, or {} for None.

resolve_agent

def resolve_agent(agent=None, *, model=None, deploy=None, strict: bool=False, name: str | None=None, runtime_overrides: dict | None=None, embodiment: str='robot') -> ResolvedAgent

Resolve the agent (a path or a merged tree; default configs/default.toml) with model roles from model (a model name, path or tree; default the agent’s own model) against deploy (a Deploy, a name, or None for $HARNESS_DEPLOY), for an embodiment kind (the bench’s: its skills and prompt). strict: a service or skill the agent names that cannot run here is an error instead of being left out. runtime_overrides (an experiment’s changes to the agent) apply after the model file’s runtime settings.

harness_policy.config.experiments

Experiments: the agent with a model on a benchmark, as configs/experiments/<name>.toml describes it.

agent = "../ablations/r2.toml"          optional: the agent file (default configs/default.toml)
model = "grok-low"                      optional: configs/models/<name>.toml (default the agent's)
bench = "robodojo"                      robodojo | robotwin
tasks = ["general_pickup"]              names, dimension names, "moving", "all"
seeds = [0]
layouts = "procedural"                  procedural (development) | eval
layout_pack_seed = 0                    eval: which official pack
deploy = "mine"                         optional; else --deploy or $HARNESS_DEPLOY
strict = true                           default: true for eval layouts
endpoint = "http://127.0.0.1:8771"      `harness program`: a worker that already runs

[budget]                                per episode, and for the whole suite
max_wall_seconds = 5400
max_cost_usd = 4.0
max_total_cost_usd = 20.0

[worker]                                options of the benchmark's worker
record_video = false

[overrides.agent]                       changes to the agent for this experiment
runtime.keep_images = 1
[overrides.model]                       changes to the model roles
main.reasoning_effort = "low"

--set key.path=value writes into this tree; --set agent.a.b=v and --set model.a.b=v go to the overrides, --set model=NAME (or --model NAME) picks the model file.

is_experiment

def is_experiment(tree: dict) -> bool

Whether a loaded file is an experiment (it names a bench, tasks, seeds, layouts or an agent) rather than an agent file.

Experiment

class Experiment

A loaded experiment: its tree (with –set items applied), the agent resolved against the deploy, the expanded tasks and seeds, and the experiment’s budget, worker and suite options. task_config gives the driver’s configuration for one task.

task_config

def task_config(task: str | None=None, seeds=None, endpoint: str | None=None)

The driver’s configuration for one task (default: the experiment’s first).

task_description

def task_description(runtime: dict, task: str) -> str

The task’s description (RoboDojo wiki recipe) when the agent uses them.

load_experiment

def load_experiment(path=None, *, tree: dict | None=None, deploy=None, sets=(), tasks=None, seeds=None, strict: bool | None=None, model=None) -> Experiment

An experiment file (or tree; neither: an empty experiment, the default agent on RoboDojo’s development layouts) with –set items applied, its agent resolved against the deploy (argument, the file’s deploy, or $HARNESS_DEPLOY). model (a model name or file) replaces the experiment’s.