pointing

Open-vocabulary pointing and 3D localization from one or more cameras.

• Layer: base
• Needs: one of pointer, vlm
• Requires: –
• Conflicts with: –
• Never dropped: no
• Mixin: skills.pointing:Pointing
• Manifest: skills/pointing/skill.toml

Functions

point_to

def point_to(observation, prompt, camera='cam_head', tiles=None)

Pixels (u, v) in observation.images[camera] that match the prompt. With tiles (default: for cam_head, where objects look small), a prompt found nowhere in the whole image is searched again in enlarged tiles.

localize

def localize(observation, prompt, camera='cam_head', z=None)

Point at prompt in one camera and intersect the pixel ray with the horizontal plane at height z (default: the table top). Returns {“pixel”: (u, v), “position”: [x, y, z]} for each match, best first. For tall objects pass the height of the point you expect the model to pick, or triangulate the same point from two cameras.

localize_3d

def localize_3d(observation, prompt, cameras=('cam_head', 'cam_left_wrist', 'cam_right_wrist'), max_residual=0.03)

3D position of a single object named by prompt, triangulated from the cameras that see it (point_to in each). Unlike localize, this does not assume a height. Returns {“position”, “cameras”, “residual_m”, “single_view”}. When several objects match the prompt, points are paired across views by geometric consistency (head camera first). When no two views agree (e.g. an arm occludes the object from the head camera), the position is one camera’s ray at 2 cm above the table and single_view is True. None when no camera sees it.

What the model reads

From the RoboDojo prompt pack (prompts/robodojo/), with every skill and feature on:

  point_to(observation, prompt, camera="cam_head", tiles=None) -> [(u, v), ...]
      pixels matching the prompt; on cam_head (or with tiles=True) a prompt found
      nowhere is searched again in enlarged tiles, for small objects. Phrasing matters:
      when nothing is found, try the color, the material, or the holder it sits in.
  localize(observation, prompt, camera="cam_head", z=None) -> [{"pixel", "position"}, ...]
      position = where the pixel ray meets the horizontal plane at height z
      (default TABLE_Z = 0.765); for tall objects pass the height of the pointed part
  localize_3d(observation, prompt) -> {"position", "cameras", "residual_m", "single_view"} or None
      triangulates one object seen by two or more cameras (head and wrists); no
      height assumption, so prefer it over localize for objects taller than a few cm.
      When several objects match, views are paired by geometric consistency. When no
      two views agree (single_view) it assumes 2 cm above the table. The arms can hide
      objects from cam_head: move an arm aside and observe again when nothing is found.