Skip to content
The HarnessLMMs-Lab docs

Loading search…

Quickstart

From pip install harness-policy (Installation) to scored episodes. The examples run the default agent on RoboDojo’s development layouts with DeepSeek V4.1 Flash; any agent, model or benchmark works the same way.

1. Run an episode

Section titled “1. Run an episode”
Terminal window
harness run dev

dev is an experiment: the default agent with its default model on one task and one development layout, with a $4 cost cap per episode. The first time, the harness installs what the agent uses and this machine lacks (RoboDojo’s environment, the SAM 3 segmenter, the Simate policy) and starts the models on the GPUs it finds; later runs start at once. It prints one line per episode, such as these two of an evaluation run (agent 1.4.1):

[slot 0] stack_bowls seed 0: episode_ended success=True reward=1.0 calls=2 cost=$0.002 wall=272s
[slot 0] general_pickup seed 4: finished success=True reward=1.0 calls=4 cost=$0.003 wall=245s

episode_ended: the learned policy, handed the task, finished it and the benchmark ended the episode; finished: a program called finish(). Results go to runs/dev.

2. See what a run resolves to

Section titled “2. See what a run resolves to”

An experiment names a benchmark, tasks and seeds, and may pick a model or a variant of the agent. harness config shows what it resolves to on this machine:

Terminal window
harness config dev
agent default 1.6.0  identity 3786ebbcd0e640ae  mode program
main model: deepseek/deepseek-v4.1-flash (openrouter)
bench robodojo, 1 tasks x 1 seeds, procedural layouts, strict False
skill         on   why
core          yes
geometry      yes
kinematics    yes
pointing      yes
vlm           yes
segmentation  yes
grasping      yes
memory        yes
verify        yes
motion        yes
manipulation  no   not in this configuration
eef           yes
reactive      yes
scene         yes
handlers      yes
vla           yes
wam           no   no world_model service
uses       service  status
segmenter  sam3
policies   simate

(manipulation, the pick and place by name, belongs to the analytic-manipulation variant; wam, the world model’s predict_future, comes with --set 'agent.services.world_models=["openwam"]'.)

The identity is the hash of everything that decides what the run computes; where the models run is not part of it.

3. Check the pipeline without a model

Section titled “3. Check the pipeline without a model”

Replay fixed model replies instead of calling a model. This starts the environment, runs a short program that lifts one arm, stops a second lift with a guard, works the grippers, runs a controller and finishes, and costs nothing:

Terminal window
harness run dev --script pipeline_check

4. Change the model, the tasks, the agent

Section titled “4. Change the model, the tasks, the agent”

harness run resumes: an episode that already has a result is not run again. Change the tasks, the seeds, the model, or anything in the agent for one run with flags:

Terminal window
harness run dev --tasks stack_bowls,stack_blocks --seeds 0-4 --model grok-medium

5. Look at the results

Section titled “5. Look at the results”
Terminal window
harness report runs/dev     # score and success rate, per task and dimension
harness view runs/dev       # every model request, program, event and action, as HTML

Every episode directory holds manifest.json (configuration, lock, identity, code fingerprint), events.jsonl (each model request and reply, each program and its events, the outcome), summary.json and kernel.log. Read a run walks through one.

Next

Section titled “Next”