Services and environments

A service is a model process that skills or model roles call: SAM 3 for segmentation, Molmo2 for pointing, a VLM, a VLA, OpenWAM, the kinematics. An environment is where tasks run: a benchmark’s official environment on this machine, or a task from The Simulator. Both are installed the first time a run needs them and never before; placement on the GPUs never changes what a run computes, only where and when.

Manifests

One file per service, services/<name>.toml: how to start it, its port and memory, the environment it runs in, and where its weights come from.

kind = "segmenter"
summary = "SAM 3 instance segmentation from text or point prompts"
protocol = "http"
command = ["{python}", "-u", "services/servers/serve_sam3.py", "--checkpoint", "{weights}/sam3/sam3.pt",
           "--port", "{port}"]
port = 8810
vram_gb = 7.2
load_seconds = 60
client = { url = "{url}" }

# The environment the service runs in, built the first time a run needs it.
[environment]
python = "3.11"
requirements = ["torch==2.7.0", "timm==1.0.30", "ftfy==6.1.1", "iopath==0.1.10", "pycocotools==2.0.11"]
sources = [{ repo = "https://github.com/facebookresearch/sam3", rev = "<pinned commit>", install = "editable" }]

[[weights]]
dir = "sam3"
modelscope_model = "facebook/sam3"

{python} is the interpreter of the service’s own environment, so services whose dependencies conflict (a policy that needs Python 3.10, another on JAX) each get theirs. A learned policy needs no server code of ours: protocol xpolicylab names the XPolicyLab policy folder, checkpoint and overrides, and XPolicyLab’s own setup_policy_server.py serves it. Any of XPolicyLab’s policies becomes a learned skill with one manifest (Add a VLA). The services and their measured VRAM are in the services reference; the manifest keys in the configuration reference.

Installed on first use

When a run starts, the agent’s configuration decides which skills are on, and each skill names the kinds of service it needs. For every service a run will call that this machine does not have, the harness builds the environment its manifest describes, installs the pinned requirements and sources, downloads and checks the weights, and then starts it. Everything lives under ~/.cache/harness (HARNESS_HOME), so the next run starts at once. A skill whose service is turned off in the configuration is never installed; it drops out of the run, and the prompt drops its text with it.

Benchmarks work the same way: the first run on RoboDojo installs its official environment (the simulator, its assets and the observation contract), and a run on a task from The Simulator needs only The Simulator’s client.

Placement and deploys

Without a deploy file the harness places what it starts on the GPUs it finds: resident when everything fits, otherwise on demand, sharing a GPU one model at a time. A deploy file, configs/deploy/<machine>.toml, chooses instead: the GPUs the project may use, where weights live, the simulator workers, and how each service runs. A deploy describes one machine, so it is not part of the project: git ignores every file in configs/deploy/ but the example-*.toml ones below, and a file elsewhere works as --deploy PATH or $HARNESS_DEPLOY.

Mode	Meaning
resident	started before the first episode (or by harness up), kept running
on_demand	started by the broker when a skill first calls it; stopped after idle_seconds or when a peer of its swap group is needed
external	someone else runs it, here or on another machine; only a url, nothing is installed
off	not here

gpu = "auto" places a service on the GPU with the most room for its vram_gb, largest first; the members of a swap group go to one GPU and need the room of the largest.

Four GPUs

# A machine with four 24 GB GPUs: the agent's models resident on GPU 0 (SAM 3 and
# Simate), Molmo2 and OpenWAM on demand in one swap group on GPU 1 for experiments that
# turn them on (a pointer role, `--set 'agent.services.world_models=["openwam"]'`), and
# the simulator workers on the other two, two per GPU.
# Copy to configs/deploy/<your machine>.toml (git-ignored) and set the paths and GPUs.
weights = "~/harness-weights"
state = "~/.cache/harness-policy/services"
gpus = [0, 1, 2, 3]
gpu_memory_gb = 24

# Simulator workers of `harness run`: two RoboDojo workers per GPU (6-9 GB each).
[sim]
gpus = [2, 3]
workers_per_gpu = 2
base_port = 8771

[services.sam3]
gpu = 0

[services.simate]
gpu = 0

[services.molmo2]
gpu = 1
mode = "on_demand"
swap = "experiments"

[services.openwam]
gpu = 1
mode = "on_demand"
swap = "experiments"

Two GPUs

# A machine with two 24 GB GPUs: the agent's models stay resident on GPU 1 (SAM 3 7.2 GB
# and Simate 14 GB; started once with `harness up`), the simulator workers run on GPU 0.
# Copy to configs/deploy/<your machine>.toml (git-ignored) and set the paths.
weights = "~/harness-weights"
state = "~/.cache/harness-policy/services"
gpus = [0, 1]
gpu_memory_gb = 24

[sim]
gpus = [0]
workers_per_gpu = 2

[services.sam3]
gpu = 1

[services.simate]
gpu = 1

One GPU

# A machine with one 24 GB GPU. The simulator (6-9 GB) leaves room for one model at a
# time: SAM 3 (7.2 GB) and Simate (14 GB) are on demand in one swap group, so the broker
# loads the one a skill calls and swaps when the other is needed. The simulator waits
# while a model loads, so this costs wall time, not episode steps, and the identity is
# that of a machine where both stay resident. One thing differs: Simate keeps a memory of
# past frames, and swapped out it starts that memory afresh, so a policy run after a SAM 3
# call in the same episode does not see the frames before it (docs/src/content/docs/guides/small-machine.md).
# Copy to configs/deploy/<your machine>.toml and set the paths.
weights = "~/harness-weights"
state = "~/.cache/harness-policy/services"
gpus = [0]
gpu_memory_gb = 24

[sim]
gpus = [0]
workers_per_gpu = 1

[services.sam3]
gpu = 0
mode = "on_demand"
swap = "models"

[services.simate]
gpu = 0
mode = "on_demand"
swap = "models"

No GPU for models

# No GPU for models: only the simulator runs here, every model is hosted (the main and
# vlm roles). Without a segmenter, segmentation and the tools built on it (locate, pick,
# release, ...) drop out of runs on this machine, and without a policy server the
# learned skills: a smaller configuration with another identity.
weights = "~/harness-weights"
gpus = [0]

[sim]
gpus = [0]
workers_per_gpu = 1

On demand and swap groups

On a machine without a GPU per model, models share GPUs. An on-demand service starts when a client first asks for it: every client (the policy and world-model clients, the segmenter, chat roles backed by a service) asks the broker before each call, and the broker starts the service and waits until it answers. Services of one swap group share a GPU, one loaded at a time; the broker waits while a peer is in use, then swaps.

The simulator does not advance while a model loads (a skill’s call simply takes longer), so swapping costs wall time, not episode steps: the run computes the same thing as with every model resident, except that a policy with memory (scoped = true, such as Simate) that is swapped out starts its memory afresh.

Commands

A run installs and starts what it needs by itself; these commands do it by hand.

harness services                       # the manifests
harness doctor                         # what is installed, what an agent would still install, GPUs and free memory
harness install sam3 simate            # install ahead of time: environments, sources, weights
harness up                             # start the resident services (and the broker)
harness status
harness down                           # stop what harness up started

harness up leaves a service alone when its port already answers (started by hand or by another checkout); harness down stops only what it started, by signal escalation (SIGINT or SIGTERM, then SIGKILL). Logs and pid files go to the deploy’s state directory. Manage services has the details.

A deploy written for you

An exported XPolicyLab policy writes its own deploy at start, for the GPUs its policy server sees: every service on ports of its own (several instances may share a machine), resident when they fit the free memory, else policies and world models on demand in one swap group, else every model but the kinematics. The agent’s models (kinematics, SAM 3, Simate: 23.2 GB) fit one 24 GB GPU resident (Evaluation through XPolicyLab).