Use a self-hosted model

Any OpenAI-compatible server can take a model role (backend = "openai_chat"), and a role can name a service of the deploy, which then supplies the URL and the model id. services/servers/serve_vlm.py serves a local vision-language model with transformers alone (one request at a time; meant for development without API spend).

The services

kind = "llm"
summary = "Qwen3-VL-32B-Instruct in 4 bit, self-hosted: the development model that writes programs (and answers VLM questions)"
protocol = "openai_chat"
command = ["{python}", "-u", "services/servers/serve_vlm.py", "--model", "{weights}/qwen3-vl-32b",
           "--name", "Qwen3-VL-32B-Instruct", "--quantize", "4bit", "--port", "{port}"]
port = 8800
# 4-bit on one 24 GB GPU; runs out of memory above about 7k prompt tokens.
vram_gb = 21.0
load_seconds = 300
client = { backend = "openai_chat", base_url = "{url}/v1", model = "Qwen3-VL-32B-Instruct" }
needs_files = ["{weights}/qwen3-vl-32b/config.json"]

[[weights]]
dir = "qwen3-vl-32b"
modelscope_model = "Qwen/Qwen3-VL-32B-Instruct"

Service	Kind	Model	VRAM
qwen3-vl-32b	llm	Qwen3-VL-32B-Instruct, 4 bit	21 GB
qwen3-vl-8b	vlm	Qwen3-VL-8B-Instruct, bf16	18 GB
molmo2	pointer	Molmo2-8B, 8 bit, vision backbone in bf16	12.7 GB

Place one in your deploy and start it:

[services.qwen3-vl-32b]
gpu = 4

harness install qwen3-vl-32b
harness up qwen3-vl-32b --deploy mine

The model file

# Self-hosted development model, no API spend: Qwen3-VL-32B-Instruct (4 bit, the
# qwen3-vl-32b service) writes the programs, answers questions and points; Molmo2 for a pointer role.
description = "Qwen3-VL-32B (self-hosted, 4 bit) writes programs, answers questions and points; Molmo2 for a pointer role"

[main]
service = "qwen3-vl-32b"
max_output_tokens = 2048
# Sampling breaks the exact-repeat loops a small model falls into at temperature 0.
temperature = 0.6
timeout_seconds = 600

[vlm]
service = "qwen3-vl-32b"
timeout_seconds = 300

[pointer]
service = "molmo2"

[runtime]
keep_images = 1
# The 32B model in 4 bit on one 24 GB GPU runs out of memory beyond about 7k prompt
# tokens; with images downscaled by the server three turns fit.
max_history_turns = 3
# The task descriptions do not fit its context either.
task_descriptions = false

harness run configs/experiments/dev.toml --model qwen3-vl-32b --output runs/local

A role’s own keys go on top of the service’s client config (temperature, timeout_seconds, max_output_tokens, extra_body). Self-hosted services report no cost, so cost_usd counts only hosted calls.

Small context windows

The 32B model in 4 bit runs out of memory above about 7k prompt tokens on a 24 GB GPU. Its model file keeps the history short: keep_images = 1, max_history_turns = 3, and no task descriptions. When a request still runs out of GPU memory, the driver drops one more turn and retries. serve_vlm.py --max-image-side N downscales images on the server.

Any other OpenAI-compatible server

[main]
backend = "openai_chat"
base_url = "http://10.0.0.9:8000/v1"    # vLLM, SGLang, a local proxy, ...
model = "Qwen/Qwen3-VL-235B-A22B-Instruct"
api_key_env = "MY_SERVER_KEY"           # optional; "EMPTY" without one