manipulation
Pick and place by name over the analytic skills (pointing, plan_grasp), and a straight-line push; an ablation of the default tools by object id (skills/eef).
- Layer: base (optional: only where a configuration enables it)
- Needs: segmenter; one of pointer, vlm
- Requires: pointing, segmentation
- Conflicts with: eef
- Never dropped: no
- Mixin:
skills.manipulation:Manipulation - Manifest:
skills/manipulation/skill.toml
Functions
Section titled “Functions”def pick(target, arm=None, attempts=3, lift=0.1, destination=None, near=None)Pick up the object named target, coarse to fine: the head camera finds it
(after moving the arms aside once when it does not), then the arm’s wrist
camera looks at it from about 25 cm and again from farther back (the outline’s
height from how much it shrinks), and the grasp is planned from that view.
Descend, close, check holding, lift, check again. A failed attempt opens,
backs off and looks again at the same object (within 10 cm of where it was).
near ([x, y] or [x, y, z]) picks, among several objects matching the name,
the one nearest that point (within 10 cm). target may also be a position
[x, y, z] (a list, tuple or numpy array, or a localize_3d result): the object
there, whatever it is.
Without arm: the arm nearer the object, or, given destination (where the
object goes next: an object name or [x, y, z]), an arm that reaches both.
An object that moves meanwhile (a conveyor) is intercepted instead: its
velocity from the two head-camera sightings, the gripper waits above a point
of its path that an arm reaches in time, follows it down and closes on it.
Returns {“ok”, “arm”, “approach”, “attempts”, “position”, “reason”, “tries”};
tries lists every attempt’s view, grasp and outcome.
def place(target, arm, clearance=0.01)Put the object held by arm on the object named target (the instance this
arm can reach, not what it holds), or at a surface position [x, y, z] (a list,
tuple or numpy array, or a localize_3d result; [x, y] is on the table). For a
named target the other arm’s wrist camera, when that arm reaches it, looks at
it from above to locate it exactly and backs off toward its base (it moves
aside fully only when that fails or the holding arm cannot get above the
target); then the object is lowered slowly toward clearance above the surface
and beyond, stopping at contact, released and the arm moves back up. The
approach starts 10 cm above the release point, or 6 or 4 cm where the wrist
cannot get that high (a stack). Nothing is released when the arm cannot get
there. Returns {“ok”, “position”, “surface”, “contact”, “below_surface_m”,
“reason”}, plus “warning” when the object met the table well below the target’s
located point: inside a container that is expected, on top of an object it means
it missed and lies beside the target. below_surface_m is how far the object’s
bottom went below the target’s located point.
def push(arm, start_xy, end_xy, height=None, speed=PUSH_SPEED, yaw=None)Push along a straight line at a fixed height: close the gripper (what it
holds stays held and pushes), move the fingertips above start_xy, lower them
to height (fingertip z; default PUSH_HEIGHT_M above the table), move them in
a straight line to end_xy at speed m/s (inverse kinematics every tick, so
the line is exact, unlike move_eef’s planned path), and lift them back up.
The fingers point down, closing across the line (yaw overrides: the closing
direction’s angle), or tilted toward the arm’s base where down is out of reach.
Returns {“ok”, “arm”, “reached” [x, y], “pushed_m”, “length_m”, “reason”}; when
the fingertips fall behind the line (the object does not move, or something
holds the arm) the push stops there: ok False, reason “blocked”, reached and
pushed_m say how far it got.
What the model reads
Section titled “What the model reads”From the RoboDojo prompt pack (prompts/robodojo/), with every skill and feature on (manipulation in place of eef):
pick(target, arm=None, attempts=3, lift=0.10, destination=None, near=None) -> {"ok", "arm", "approach", "attempts", "position", "reason"} closed-loop pick of the object named target: localizes it (clear_view once if nothing is found), plans a grasp, re-plans from the wrist camera at the pregrasp, descends, closes, checks holding, lifts and checks again; retries from a fresh observation on failure. Among several alike objects pass near=[x, y] (from localize_3d) to take the one nearest that point (within 10 cm); target may also be a position [x, y, z] (list or numpy array): the object there. Pass destination (the object name or [x, y, z] it goes to next) to get an arm that reaches both. An object that moves (a conveyor) is intercepted: the gripper waits above its path and follows it down; call pick as soon as the object is in view, since it may leave the reach. place(target, arm, clearance=0.01) -> {"ok", "position", "surface", "contact", "below_surface_m", "reason"} puts the held object on the object named target, or at a surface position [x, y, z] (list or numpy array; [x, y] = on the table): moves above (from 10 cm, or 6 or 4 cm over a stack the wrist cannot clear higher), has the other arm locate a named target up close when that arm reaches it, lowers the object's bottom to clearance above the surface (contact stops the descent), opens and moves up. ok is False when it cannot get there (still holding). A "warning" says the object met the table more than 2 cm below the target's located point: expected inside a container, but on a stack it missed and lies beside the target (look, then pick it again). contact False means nothing stopped the object down to 3 cm below the target: right for a container, suspect on a stack. push(arm, start_xy, end_xy, height=None, speed=0.05, yaw=None) -> {"ok", "reached", "pushed_m", "length_m", "reason"} closes the gripper (a held tool stays held and pushes), lowers the fingertips to `height` (default 1.5 cm above the table) at start_xy and moves them in an exact straight line to end_xy, then lifts. Use it to slide objects that must not be lifted (push T, align blocks, sweeping): start a few cm behind the object's edge, on the side away from end_xy. A push that stalls stops: reason "blocked", reached and pushed_m say how far it got.