segmentation

Instance masks from a text prompt or a pixel (SAM 3).

• Layer: base
• Needs: segmenter
• Requires: –
• Conflicts with: –
• Never dropped: no
• Mixin: skills.segmentation:Segmentation
• Manifest: skills/segmentation/skill.toml

Functions

segment_text

def segment_text(observation, prompt, camera='cam_head', threshold=None)

Instance masks for prompt in observation.images[camera], best first. By default keeps instances scoring at least half the best score.

segment_point

def segment_point(observation, xy, camera='cam_head', labels=None)

Masks of the object under pixel xy (or a list of pixels) in camera, best first. labels: 1 = on the object (default), 0 = background, one per pixel.

What the model reads

From the RoboDojo prompt pack (prompts/robodojo/), with every skill and feature on:

  segment_text(observation, prompt, camera="cam_head") -> [Mask]   instance masks, best first, or []
      Mask has .array (HxW bool), .box, .score, .centroid (u, v), .pixels(). Scores
      depend on phrasing: describe color and object ("white bowl") or use a plural.
  segment_point(observation, (u, v), camera="cam_head", labels=None) -> [Mask]
      masks of the object under a pixel (or a list of pixels; labels 1 = object,
      0 = background), best first; use it on a pixel from point_to when segment_text
      finds nothing.