Conversations

harness.runtime.decision.model

Multi-turn text completion for the high level.

Chat Completions covers OpenRouter and self-hosted OpenAI-compatible servers (vLLM, SGLang) through openai_chat with a base_url. Messages use the harness content parts (input_text, input_image).

add_usage

def add_usage(total: dict, usage: dict) -> None

Add usage (one call’s or a total) to total, key by key of USAGE.

ChatReply

class ChatReply

One completion: its text, usage (USAGE and finish_reason; cost_usd None when the provider does not report it), the provider’s response id, and the seconds the request took with its retries.

valid_effort

def valid_effort(value)

effort_fields

def effort_fields(backend: str, base_url, model: str, effort, max_tokens: int) -> dict

The request fields that set effort (none, low, medium, high) for this provider, each in the field its API reads; {} when unset. OpenAI, Gemini, Ollama and other OpenAI-compatible servers take OpenAI’s reasoning_effort.

chat_url

def chat_url(provider, base_url)

The Chat Completions URL of a model config (backend, base_url).

chat_part

def chat_part(part)

marks_cache

def marks_cache(model: str) -> bool

package_transport

def package_transport(config: dict)

The transport of a backend a package provides (extensions.py model_backend).

ChatModel

class ChatModel
    __init__(config: dict, transport=None)

A Chat Completions client for one model role (MODEL_KEYS): OpenRouter, an OpenAI-compatible server, or a backend of a package, which answers the same request (package_transport). The openrouter backend needs its key in the environment; transport replaces the HTTP call (tests).

payload

def payload(system: str, messages: list[dict]) -> dict

The request body: system and the harness messages as chat messages, the output limit and extra_body fields, and on OpenRouter usage accounting and reasoning.

close

def close() -> None

Let go of what the transport holds (a package backend’s process; nothing for HTTP).

streams_text

def streams_text() -> bool

Whether complete() can hand over the reply’s text as it streams (on_text).

complete

def complete(system: str, messages: list[dict], on_text=None, *, cancellation=None) -> ChatReply

One completion of messages under system, starting an on-demand service through the broker first. Failed requests are retried (retries times, or until retry_seconds have passed), except client errors other than HTTP 429; the last failure is raised as a RuntimeError. When it streams (streams_text), on_text(kind, text) gets each piece of the reply (“text”) and of its reasoning (“reasoning”) as it comes. A cancellation closes built-in transports and stops retries.

urllib_transport

def urllib_transport(url, headers, payload, timeout, *, cancellation=None)

POST a chat completion and return the provider’s response.

sse_transport

def sse_transport(url, headers, payload, timeout, on_text=None, *, cancellation=None)

POST a streamed chat completion and assemble the chunks into the non-streamed response shape. timeout bounds each read, so keep-alive comments reset it. on_text(kind, text) gets each piece as it comes (ChatModel.complete).

scripted_models

def scripted_models(script) -> dict

The models of a scripted episode: script is a list of replies, or {“replies”: […], “verifier”: […]} as scripts/replay_script.py writes it; without recorded verifier answers the verifier says YES.

still_running

def still_running(messages: list[dict]) -> bool

Whether the last thing the request tells, before how things stand now (its ambient part), is that a program still runs and nothing reached the model: it was just started, or a wait’s time is over while units run (a wait whose time is over with nothing running is the script’s own to answer).

ScriptedChat

class ScriptedChat
    __init__(replies: list[str], waits: bool=True)

Replays fixed replies in order. For pipeline checks without model calls. A turn that comes back while a program still runs is answered <wait> without taking a reply (waits counts them), so the script is the replies after each program’s result; with waits=False every call takes the next reply.

complete

def complete(system: str, messages: list[dict]) -> ChatReply

The next scripted reply (a <finish> block once they run out), with zero usage.

harness.runtime.task.run

One conversation with one model: turns in it, model decisions and Automatic Control.

A conversation starts with one model. Every program the model writes runs as a unit of its own in the kernel (runtime/execution/kernel.py): the model’s call comes back at once, with the program’s result when it was over within a short grace, else with its handle, and the result reaches the model when the unit ends. What reaches the model by itself (a message, an event, a unit’s end, a note of the host) is a notice in one queue with a priority (runtime/decision/notices.py): now sets a reply made before it aside, next is told with the next call, later after the model’s next wait. One place tells the model with each call (_deliver): what its last reply came to, the notices, and how things stand now.

A turn is the model’s work from what started it (the instruction, a message, a unit’s end, an event) to its answer: a reply that is text, with no program and no wait (<finish> and a program’s finish() are that same answer). A limit (model calls and wall time, both of the turn; cost, of the conversation), an error, an interrupt and the environment’s end end a turn too. Every turn ends with a result record (decisions.TURN_ENDS); then the conversation is IDLE and keeps its messages, the kernel’s namespace, rules, facts and the environment. The units still running after an answer go on, and what reaches the model starts the next turn. Only the user or the host ends the conversation (close()), or a kernel that died; a caller without a user (interactive False: a benchmark’s run) ends it after the first turn.

A conversation with a user writes what continues it in another process as it goes (conversation.json beside its trace: its messages, what the user said, its facts): a TaskRun given that (restore: saved()) goes on in the same directory and trace with a new kernel, and tells the model what died with the old process.

saved

def saved(directory) -> dict | None

What the conversation in directory kept to be continued (TaskRun’s restore): {“messages”, “said”, “facts”, “units”, “turns”, “summary”}; None when it kept nothing.

TaskRun

class TaskRun
    __init__(config, seed: int, directory, *, chat=None, verifier=None, environment=None, interactive: bool=False, confirmation: str='never', instruction: str | None=None, agents=None, parent=None, memory=None, restore: dict | None=None)

One episode of config.task with seed, its trace and files under directory. chat, verifier and environment replace the configured models and worker. interactive: a user takes part (see the module docstring); confirmation: one of CONFIRMATION_POLICIES; instruction: the user’s own instruction in place of the task’s (the evaluator’s outcome still scores the task). agents (runtime/task/delegation.py) starts the sub-agents its programs ask for; parent: the agent that started this one. restore: what an earlier process kept of this conversation (saved(directory)): it goes on from there, in the same directory and trace. Managed workspaces are acquired by the host before the conversation starts.

send_message

def send_message(text: str, sender: str='user', files=()) -> None

A message from the user (or, sender “parent”, from the agent that started this one): recorded now, told with the model’s next call (a notice of priority next: the call in flight goes on; cancel() is what ends a turn). files: paths the user attached (read now: messages.attachment_parts).

pause

def pause() -> None

Pause: every unit of the model stops at its next execution step or wait, and nothing of it acts; the model is not called again until resume() or a message.

resume

def resume() -> None

Go on after a pause.

set_model

def set_model(model: str | None=None, reasoning_effort: str | None=None) -> dict

Change the main model (another id of its provider) or its reasoning effort (EFFORTS, “” for the provider’s default; runtime.thinking still bounds it) from its next call on: a model_changed record. Returns the model and effort now.

waiting_for

def waiting_for() -> str | None

What the conversation waits for the user to approve, when it is physical input.

approval_id

def approval_id()

confirm

def confirm(accept: bool, reason: str='', remember: str | None=None, *, request_id=None) -> None

Answer a pending approval. A sensitive step requires its exact request ID.

cancel

def cancel() -> None

End the turn that runs: the call in flight is set aside, the model’s units are cancelled, and the conversation waits for the next message (nothing between turns).

close

def close() -> None

End the conversation: at once while it waits, else at the next step boundary.

offer_managed_worker_approval

def offer_managed_worker_approval(workspace, reply, tick)

Publish the private worker’s attributed refusal through the normal host prompt.

worker_approval_ready

def worker_approval_ready(pending)

claim_managed_worker_retry

def claim_managed_worker_retry(pending, episode_id, step)

cancel_worker_approval

def cancel_worker_approval(pending, *, reason)

start

def start() -> threading.Thread

Run the episode on a thread of its own and return the thread.

enable_agents

def enable_agents() -> bool

The user switches sub-agents on for this conversation (any frontend): their calls load before its next program.

tell

def tell(content, source: str | None=None) -> None

The environment tells the model something that happened by itself: a message, in the form conversations give content (a string, or a list of text and image parts: decision/messages.py content_parts). It reaches the model at once (a notice of priority now): a wait ends, a call in flight is set aside and asked again; the model reads it as “Message from the environment”. Called by whoever knows of it: the run’s worker through its listener (runtime/task/inbound.py), or the code that owns this run.

queue_library_revision

def queue_library_revision(item, proof)

A trusted host verifier has admitted this exact revision.

charge_up

def charge_up() -> None

Charge what this sub-agent spent since the last time to its parent (and up).

instructions_attached

def instructions_attached(root) -> None

An environment with a folder of its own joined the conversation: the model is told that folder’s project instruction files (not the user’s again) with its next call.

seconds_left

def seconds_left() -> float

What the turn has left of its wall time (max_wall_seconds): the time it ran, less the time it waited for the user, counts; the whole budget before it starts.

run

def run() -> dict

Run the conversation to its end and return its summary (the status of its last turn, evaluator outcome, model calls, tokens, cost, times), also written with the trace and manifest under the directory at every turn’s end. An error ends the turn (before the first one: the conversation) with status “error” instead of raising.

harness.core.spans

Spans: when each part of an episode worked, for the timeline of a run.

A span is a stretch of wall-clock time one part spent on one thing: the main model answering, a program running, a skill’s bounded execution, one call to a service. It starts with a span_start record and ends with a span_end record of the same id. A span of code an entry runs (runtime/execution/entries.py) names that entry (entry); the entries themselves start and end with entry_start and entry_end records. Parts of the host (the conversation) write them to the trace themselves; code in the kernel process writes them through the sink the kernel installs (runtime/execution/kernel.py), and the host adds them to the trace as they arrive. Without a sink a span costs nothing and records nothing.

A span’s lane names the kind of part that worked (PARTS). The timeline does not draw one lane per kind: core/lanes.py reads the lanes from the records, one per entry (and the model, the verifier, the environment), as they start and end.

note

def note(kind: str, **detail) -> None

Send the host a note of kind (a plan the user sees) now, while the program runs; nothing without a sink.

step

def step(skill: str, number: int, what: str, ok: bool, seconds: float | None=None, reason: str | None=None, of: int | None=None, library=None) -> None

Say a step of a skill’s plan ended (step number of of, what it did, whether it did it, in how many seconds, why not): a skill_step record, news for the model’s next look while the program runs (runtime/conversation.py); nothing without a sink. A step of a library entry’s test or replay says so (testing, set by testing()).

in_test

def in_test()

testing

def testing(active=True)

Keep proposal and replay steps out of practice use counts.

record

def record(op: str, **detail) -> None

Send the host a record op other than a span’s (an entry’s start or end, with detail) now; nothing without a sink.

set_entry

def set_entry(entry)

The spans this thread records from now on name entry (an entry id; None: none); the one they named before.

current_entry

def current_entry()

connected

def connected() -> bool

Whether notes reach a host (a kernel installed the sink).

set_sink

def set_sink(sink) -> None

Send every span record from now on to sink(record) (None: record nothing).

current

def current()

The id of the span this thread is in (None outside every span).

under

def under(parent)

Spans recorded inside have parent (a span id from current(), or None) as their parent: code another thread runs for a span (the tick runs a skill’s steps).

span

def span(lane: str, name: str, **detail)

Record the code inside as a span of the part lane (PARTS) named name with detail. Yields a dict whose items end up in the span’s end record (an event kind, a status); an exception leaving the block is recorded as the span’s error and raised on.