Perceiving Objects Oneshot

perceiving-objects-oneshot

Single VLM call over a set-of-marks overlay. Pipeline:

observe → perceive → filter_obb

perceive runs:

1. `grounding-dino.detect with a broad object.` text prompt
2. One `vlm.query` showing the image with letter-labeled boxes:

"Which letter is the **? Reply with one letter or 'none'."

1. On `none: emit found: False` so the subgraph exits

`not_found. On a letter: sam3.segment_box on the chosen box, geometry.mask_to_world_points` for the cloud.

When to use

• Clean-all-items / multi-item loops where the cycle needs a clean

"no match" signal to terminate via `target.not_found → done`.

• Tasks where the target description is generic ("any item on the

floor", "the next remaining grocery item") rather than a specific scene-spec id.

• Uncluttered scenes with distinct, reasonably sized targets where the

set-of-marks letter pick is reliable.

When NOT to use

• Small / cluttered targets ( About object_name below: it is a literal Python string — the

> natural noun phrase for the object you are perceiving, drawn from this > subgraph's description (e.g. "alphabet soup", "basket", > "any grocery item on the floor"). It is a constant per subgraph > instance, NOT a binding. DO NOT write Ref("in.object_name") or > any other Ref(...); the coordinator does not declare object_name > as a subgraph input. Write the string directly, > e.g. "object_name": "any grocery item on the floor". > The same rule applies to object_description if you set it.

1. observe — type: tool, tool: "robot.get_observation",

inputs: {}. Connector tool; flat name only.

1. perceive — type: script, file scripts//perceive_simple.py

from this bundle. Inputs: cameras=Ref("observe.cameras"), object_name="", plus any optional literals (object_description, dino_prompt). Returns {found, cloud, mask, score}. When the VLM picks "none" or DINO emits no detections, `found is False and the downstream filterobb step then raises (empty cloud) — caught by the subgraph's onerror: "not_found"` exit.

1. filter_obb — type: tool,

tool: "geometry.filter_and_compute_obb", inputs={"points": Ref("perceive.cloud")}. Returns {"obb": }.

Wiring the exit (HARD)

Linear perceive → filter_obb → found → END. The filter_obb tool raises on empty clouds (the not-found path), and the subgraph's on_error: "not_found" catches that. Do NOT add conditional edges on perceive — the linear path plus set_on_error is sufficient.

sg.add_node("filter_obb", type="tool",
            tool="geometry.filter_and_compute_obb",
            inputs={"points": Ref("perceive.cloud")})
sg.add_exit("found")
sg.add_edge("perceive", "filter_obb")
sg.add_edge("filter_obb", "found")
sg.add_edge("found", END)
sg.set_on_error("not_found")

Bind the subgraph outputs (ALL THREE — required, no exceptions):

sg.set_outputs(
    target_obb=Ref("filter_obb.obb"),
    target_mask=Ref("perceive.mask"),
    target_cloud=Ref("perceive.cloud"),
)

Note that geometry.filter_and_compute_obb returns {"obb": ...}, so the OBB binding walks into the obb field (Ref("filter_obb.obb"), NOT a bare Ref("filter_obb")). See references/geometry_calling_conventions.md.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: graph-robots
• Source: graph-robots/open-robot-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.