Annotate Reality

Annotate Reality

Anchor information to the physical world. The perception-agent decides what a real thing is; you render the spatial label/box/pin in the right place.

Pick the annotation

| Goal | Widget | Tool | | ---- | ------ | ---- | | Name a real object with a callout | vision_annotation | annotate_object | | Box a detected object's extent | bounding_box_3d | draw_bounding_box | | Drop a pin/marker at a spot | scene_label | drop_scene_label | | Free-floating heading/caption | text_label | show_text |

Steps

1. Get the target position (world meters) from the perception annotation,

gaze hit point, or a scene surface.

1. Choose the widget from the table; keep labels short.
2. Anchor in world with billboard:true so the label faces the user; lift

the callout slightly above the object (e.g. +0.15 m in Y) with a leader_line.

1. Spawn, then keep it anchored as the user moves (world anchor handles this).
2. Clean up stale annotations via close_hologram when no longer relevant.

Output

vision_annotation (annotate_object, props per registry.json):

{ "widget_type": "vision_annotation",
  "transform": { "anchor": "world", "position": [0.3,0.95,0.7], "billboard": true },
  "props": { "label": "coffee mug", "confidence": 0.78, "detail": "ceramic, ~350 ml",
             "leader_line": true, "target_position": [0.3,0.8,0.7], "color": "#7FE7FF" },
  "interactions": ["tap","grab","dwell"] }

bounding_box_3d (draw_bounding_box):

{ "widget_type": "bounding_box_3d",
  "props": { "label": "laptop", "confidence": 0.91, "size": [0.33,0.02,0.23], "color": "#FFB74D" } }

scene_label (drop_scene_label):

{ "widget_type": "scene_label",
  "props": { "text": "Keys (last seen here)", "icon": "pin", "color": "#FF5252", "pin": true } }

Edge cases

• No reliable position → fall back to a head-anchored text_label and say

you couldn't pin it precisely.

• Many annotations → cap to the relevant few; over-labeling clutters reality

(hand to declutter-space).

• Moving target → annotations are static once placed; re-place if the object

moves.

• Confidence honesty → carry the real confidence so the UI can style

uncertainty.

• Steps/tour → sequence pins and advance as the user completes each.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: sumitaich1998
• Source: sumitaich1998/jarvisvr
• License: MIT

Install and usage instructions live in the source repository linked above.