使用自托管模型

任何与 OpenAI 兼容的服务器都可以承担一个模型角色（backend = "openai_chat"），角色也可以指定 deploy 中的一个服务，由该服务提供 URL 和模型 id。services/servers/serve_vlm.py 只用 transformers 提供一个本地视觉语言模型（一次处理一个请求；用于不花 API 费用的开发）。

服务

kind = "llm"
summary = "Qwen3-VL-32B-Instruct in 4 bit, self-hosted: the development model that writes programs (and answers VLM questions)"
protocol = "openai_chat"
command = ["{python}", "-u", "services/servers/serve_vlm.py", "--model", "{weights}/qwen3-vl-32b",
           "--name", "Qwen3-VL-32B-Instruct", "--quantize", "4bit", "--port", "{port}"]
port = 8800
# 4-bit on one 24 GB GPU; runs out of memory above about 7k prompt tokens.
vram_gb = 21.0
load_seconds = 300
client = { backend = "openai_chat", base_url = "{url}/v1", model = "Qwen3-VL-32B-Instruct" }
needs_files = ["{weights}/qwen3-vl-32b/config.json"]

[[weights]]
dir = "qwen3-vl-32b"
modelscope_model = "Qwen/Qwen3-VL-32B-Instruct"

服务	类型	模型	显存
qwen3-vl-32b	llm	Qwen3-VL-32B-Instruct，4 bit	21 GB
qwen3-vl-8b	vlm	Qwen3-VL-8B-Instruct，bf16	18 GB
molmo2	pointer	Molmo2-8B，8 bit，视觉主干为 bf16	12.7 GB

在你的 deploy 中放置其中一个并启动它：

[services.qwen3-vl-32b]
gpu = 4

harness install qwen3-vl-32b
harness up qwen3-vl-32b --deploy mine

模型文件

# Self-hosted development model, no API spend: Qwen3-VL-32B-Instruct (4 bit, the
# qwen3-vl-32b service) writes the programs, answers questions and points; Molmo2 for a pointer role.
description = "Qwen3-VL-32B (self-hosted, 4 bit) writes programs, answers questions and points; Molmo2 for a pointer role"

[main]
service = "qwen3-vl-32b"
max_output_tokens = 2048
# Sampling breaks the exact-repeat loops a small model falls into at temperature 0.
temperature = 0.6
timeout_seconds = 600

[vlm]
service = "qwen3-vl-32b"
timeout_seconds = 300

[pointer]
service = "molmo2"

[runtime]
keep_images = 1
# The 32B model in 4 bit on one 24 GB GPU runs out of memory beyond about 7k prompt
# tokens; with images downscaled by the server three turns fit.
max_history_turns = 3
# The task descriptions do not fit its context either.
task_descriptions = false

harness run configs/experiments/dev.toml --model qwen3-vl-32b --output runs/local

角色自己的键叠加在服务的客户端配置之上（temperature、timeout_seconds、max_output_tokens、extra_body）。自托管服务不报告费用，因此 cost_usd 只计算托管调用。

较小的上下文窗口

4 bit 的 32B 模型在 24 GB GPU 上，prompt 超过大约 7k token 时会显存不足。它的模型文件让历史保持简短：keep_images = 1、max_history_turns = 3，并且不提供任务描述。如果某个请求仍然耗尽 GPU 显存，driver 会再丢弃一轮历史并重试。serve_vlm.py --max-image-side N 在服务器端缩小图像。

其他任何与 OpenAI 兼容的服务器

[main]
backend = "openai_chat"
base_url = "http://10.0.0.9:8000/v1"    # vLLM, SGLang, a local proxy, ...
model = "Qwen/Qwen3-VL-235B-A22B-Instruct"
api_key_env = "MY_SERVER_KEY"           # optional; "EMPTY" without one