跳转到内容

正在加载搜索…

在 RoboTwin 2.0 上运行

RoboTwin 2.0(SAPIEN)是第二个 benchmark:50 个双臂任务,每个任务由自己的 worker(benches/robotwin/runtime.py)提供,机器人 API 与 RoboDojo 的相同。

  1. 按 third_party/sources.toml 中的 revision 检出 RoboTwin(检出到 third_party/RoboTwin,或设置 HARNESS_ROBOTWIN_ROOT),并获取其资源(Hugging Face 数据集 TianxingChen/RoboTwin2.0;sha256 值在 sources.toml 中)。
  2. bash environment/setup.sh --only robotwin:在项目环境中安装 SAPIEN 3.0.1、mplib、toppra,应用 RoboTwin 的两个源码补丁,并在 third_party/curobo-0.7.8 中就地构建 RoboTwin 自己的 cuRobo 0.7.8(environment/robotwin_curobo.sh;HARNESS_ROBOTWIN_CUROBO 可指向其他位置)。worker 会把它放在 PYTHONPATH 的最前面。

只有 RoboTwin 的专家能解决的 seed 才是有效的 RoboTwin seed。episode 索引 n 是第 n 个有效 seed:评测 seed 从 100000 × (1 + group) 开始(官方的那些),开发 seed 从 10,000,000 + 100000 × group 开始。有效 seed 缓存在 seed_cache_dir/<split>-<group>/<task_config>/<task>.json 中(seed_cache_dir 默认为 <RoboTwin>/harness_seeds)。在运行文件中,layouts = "procedural" 选择开发 split,layout_pack_seed 选择 group。

# RoboTwin 2.0 development check on stack_blocks_two: the agent with the RoboTwin prompt
# pack (AgileX ALOHA). No learned skill for this embodiment (no policy or world model
# trained on RoboTwin is served), and RoboTwin has no task descriptions.
description = "the agent on RoboTwin, development seeds"
bench = "robotwin"
tasks = ["stack_blocks_two"]
seeds = [0]
layouts = "procedural"
endpoint = "http://127.0.0.1:8772"
[budget]
max_wall_seconds = 5400
max_cost_usd = 4.0
[overrides.agent]
prompt = "../../prompts/robotwin"
# The tools by object id (skills/eef) are tuned to RoboDojo's ARX X5 (table height,
# parking spots, jaw width); RoboTwin's ALOHA uses the manipulation skill's.
skills = { layers = { learned = false }, enable = ["manipulation"], disable = ["eef"] }
services = { policies = [], world_models = [] }
# RoboTwin has no XPolicyLab embodiment profile here: the worker shows programs its own
# observations (the default agent's contract is RoboDojo's leaderboard protocol).
runtime = { task_descriptions = false, contract = "native" }
Terminal window
harness run configs/experiments/robotwin-dev.toml --deploy mine --output runs/robotwin-dev
harness run configs/experiments/robotwin-dev.toml --deploy mine --tasks all --seeds 0-9 --output runs/robotwin-all

或者手动针对一个 worker 运行:

Terminal window
harness serve --factory benches.robotwin.runtime:create_environment \
--options configs/workers/robotwin_runtime_dev.toml --port 8772
harness program configs/experiments/robotwin-dev.toml
RoboDojo RoboTwin
控制频率 25 Hz 10 Hz
指尖偏移 0.150 m 0.12 m
finish 之后机械臂回到起始位姿 是(任务要求如此) 否
学习型 skill Simate(agent 使用的)、G0.5、其他 XPolicyLab 策略;OpenWAM 用于实验 不提供
操作工具 按物体 id(locate、pick、release 等) 操作 skill 的按名称的 pick 和 place
prompt pack prompts/robodojo prompts/robotwin(自己的 core.md,其他部分共享)
自我改进脚本 有 无(它启动的是 RoboDojo worker)

该实验用 RoboTwin prompt pack 运行 agent([overrides.agent]):关闭学习层(因此没有 strategy 步骤),用操作 skill 代替按物体 id 的工具(那些工具是针对 RoboDojo 的 ARX X5 调校的),使用 SAM 3,不提供任务描述,agent 的 30 次模型调用。