# Robot Perception > > - **Type:** Skill - **Install:** `agentstack add skill-arpitg1304-robotics-agent-skills-robot-perception` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [arpitg1304](https://agentstack.voostack.com/s/arpitg1304) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [arpitg1304](https://github.com/arpitg1304) - **Source:** https://github.com/arpitg1304/robotics-agent-skills/tree/main/skills/robot-perception ## Install ```sh agentstack add skill-arpitg1304-robotics-agent-skills-robot-perception ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Robot Perception Skill ## When to Use This Skill - Setting up and configuring camera, LiDAR, or depth sensors - Building RGB, depth, or point cloud processing pipelines - Calibrating cameras (intrinsic, extrinsic, hand-eye) - Implementing object detection, segmentation, or tracking for robots - Fusing data from multiple sensor modalities - Streaming sensor data with proper threading and buffering - Synchronizing multi-sensor rigs - Deploying perception models on robot hardware (GPU, edge) - Debugging perception failures (latency, dropped frames, misalignment) ## Sensor Landscape ### Sensor Types and Characteristics ``` Sensor Type Output Range Rate Best For ───────────────────────────────────────────────────────────────────────── RGB Camera (H,W,3) uint8 ∞ 30-120Hz Object detection, tracking, visual servoing Stereo Camera (H,W,3)+(H,W,3) 0.3-20m 30-90Hz Dense depth from passive stereo Structured Light (H,W) float + RGB 0.2-10m 30Hz Indoor manipulation, short range ToF Depth (H,W) float + RGB 0.1-10m 30Hz Indoor, medium range LiDAR (spinning) (N,3) or (N,4) 0.5-200m 10-20Hz Outdoor navigation, mapping LiDAR (solid-st.) (N,3) 0.5-200m 10-30Hz Automotive, outdoor IMU (6,) or (9,) N/A 200-1kHz Orientation, motion estimation Force/Torque (6,) float N/A 1kHz+ Contact detection, force control Tactile (H,W) or (N,3) Contact 30-100Hz Grasp quality, texture Event Camera Events (x,y,t,p) ∞ μs High-speed tracking, HDR scenes ``` ### Common Sensor Hardware ``` Device Type SDK/Driver ROS2 Package ────────────────────────────────────────────────────────────────────────── Intel RealSense Structured Light pyrealsense2 realsense2_camera Stereolabs ZED Stereo + IMU pyzed zed_wrapper Luxonis OAK-D Stereo + Neural depthai depthai_ros FLIR/Basler Industrial RGB PySpin/pypylon spinnaker_camera_driver Velodyne Spinning LiDAR velodyne_driver velodyne Ouster Spinning LiDAR ouster-sdk ros2_ouster Livox Solid-state LiDAR livox_sdk livox_ros2_driver USB Webcam RGB OpenCV VideoCapture usb_cam / v4l2_camera ``` ## Camera Models and Calibration ### Pinhole Camera Model ``` 3D World Point (X, Y, Z) | [R | t] — Extrinsic (world → camera) | Camera Point (Xc, Yc, Zc) | K — Intrinsic (camera → pixel) | Pixel (u, v) K = [ fx 0 cx ] fx, fy = focal lengths (pixels) [ 0 fy cy ] cx, cy = principal point [ 0 0 1 ] Projection: [u, v, 1]^T = K @ [R | t] @ [X, Y, Z, 1]^T ``` ### Intrinsic Calibration ```python import cv2 import numpy as np from pathlib import Path class IntrinsicCalibrator: """Camera intrinsic calibration using checkerboard pattern""" def __init__(self, board_size=(9, 6), square_size_m=0.025): self.board_size = board_size self.square_size = square_size_m # Prepare object points (3D coordinates of checkerboard corners) self.objp = np.zeros((board_size[0] * board_size[1], 3), np.float32) self.objp[:, :2] = np.mgrid[ 0:board_size[0], 0:board_size[1] ].T.reshape(-1, 2) * square_size_m def collect_calibration_images(self, camera, num_images=30, min_coverage=0.6): """Collect calibration images with good spatial coverage. IMPORTANT: Move the board to cover all regions of the image, including corners and edges. Tilt the board at various angles. Bad coverage = bad calibration, especially at image edges. """ obj_points = [] img_points = [] coverage_map = np.zeros((4, 4), dtype=int) # Track board positions while len(obj_points) 0).sum() / coverage_map.size if coverage 1.0: print(f"WARNING: High reprojection error ({ret:.3f} px). " f"Check image quality and board detection.") # Compute per-image reprojection errors errors = [] for i in range(len(obj_points)): projected, _ = cv2.projectPoints( obj_points[i], rvecs[i], tvecs[i], K, dist) err = cv2.norm(img_points[i], projected, cv2.NORM_L2) err /= len(projected) errors.append(err) print(f"Calibration complete:") print(f" RMS reprojection error: {ret:.4f} px") print(f" Per-image errors: mean={np.mean(errors):.4f}, " f"max={np.max(errors):.4f}") print(f" Focal length: fx={K[0,0]:.1f}, fy={K[1,1]:.1f}") print(f" Principal point: cx={K[0,2]:.1f}, cy={K[1,2]:.1f}") return CalibrationResult( camera_matrix=K, dist_coeffs=dist, rms_error=ret, image_size=image_size) def save(self, result, path): """Save calibration to YAML (OpenCV-compatible format)""" fs = cv2.FileStorage(str(path), cv2.FILE_STORAGE_WRITE) fs.write("camera_matrix", result.camera_matrix) fs.write("dist_coeffs", result.dist_coeffs) fs.write("image_width", result.image_size[0]) fs.write("image_height", result.image_size[1]) fs.write("rms_error", result.rms_error) fs.release() @staticmethod def load(path): """Load calibration from YAML""" fs = cv2.FileStorage(str(path), cv2.FILE_STORAGE_READ) K = fs.getNode("camera_matrix").mat() dist = fs.getNode("dist_coeffs").mat() w = int(fs.getNode("image_width").real()) h = int(fs.getNode("image_height").real()) fs.release() return CalibrationResult( camera_matrix=K, dist_coeffs=dist, image_size=(w, h), rms_error=0.0) ``` ### Extrinsic Calibration (Camera-to-Camera, Camera-to-LiDAR) ```python class ExtrinsicCalibrator: """Compute transform between two sensors using shared targets""" def calibrate_stereo(self, calib_left, calib_right, obj_points, img_points_left, img_points_right, image_size): """Stereo calibration: find relative pose between two cameras""" ret, K1, d1, K2, d2, R, T, E, F = cv2.stereoCalibrate( obj_points, img_points_left, img_points_right, calib_left.camera_matrix, calib_left.dist_coeffs, calib_right.camera_matrix, calib_right.dist_coeffs, image_size, flags=cv2.CALIB_FIX_INTRINSIC # Use pre-calibrated intrinsics ) print(f"Stereo calibration RMS: {ret:.4f} px") print(f"Baseline: {np.linalg.norm(T):.4f} m") return StereoCalibration(R=R, T=T, E=E, F=F, rms_error=ret) def calibrate_camera_to_lidar(self, camera_points_2d, lidar_points_3d, K, dist): """Find camera-to-LiDAR transform using corresponding points. Use a calibration target visible to both sensors (e.g., checkerboard with reflective tape corners). """ # PnP: find pose of 3D points relative to camera success, rvec, tvec = cv2.solvePnP( lidar_points_3d, camera_points_2d, K, dist, flags=cv2.SOLVEPNP_ITERATIVE ) if not success: raise CalibrationError("PnP failed — check point correspondences") R, _ = cv2.Rodrigues(rvec) T_camera_lidar = np.eye(4) T_camera_lidar[:3, :3] = R T_camera_lidar[:3, 3] = tvec.flatten() # Verify by reprojecting projected, _ = cv2.projectPoints( lidar_points_3d, rvec, tvec, K, dist) error = np.mean(np.linalg.norm( camera_points_2d - projected.reshape(-1, 2), axis=1)) print(f"Camera-LiDAR reprojection error: {error:.2f} px") return T_camera_lidar ``` ### Hand-Eye Calibration (Camera-to-Robot) ```python class HandEyeCalibrator: """Solve AX = XB for camera mounted on robot end-effector (eye-in-hand) or camera mounted on a fixed base (eye-to-hand). Requires moving the robot to multiple poses while observing a fixed calibration target. """ def __init__(self, K, dist, board_size=(9, 6), square_size=0.025): self.K = K self.dist = dist self.board_size = board_size self.square_size = square_size self.objp = np.zeros((board_size[0] * board_size[1], 3), np.float32) self.objp[:, :2] = np.mgrid[ 0:board_size[0], 0:board_size[1] ].T.reshape(-1, 2) * square_size def collect_poses(self, camera, robot, num_poses=20): """Collect camera-target and robot poses at multiple configurations. IMPORTANT: Move to diverse robot orientations. At least 3 different rotation axes. Pure translations are NOT sufficient. """ R_gripper2base = [] t_gripper2base = [] R_target2cam = [] t_target2cam = [] for i in range(num_poses): input(f"Move robot to pose {i+1}/{num_poses}, press Enter...") # Get robot end-effector pose ee_pose = robot.get_ee_pose() # 4x4 homogeneous matrix R_gripper2base.append(ee_pose[:3, :3]) t_gripper2base.append(ee_pose[:3, 3]) # Detect calibration target in camera frame = camera.capture() gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) found, corners = cv2.findChessboardCorners( gray, self.board_size, None) if not found: print(f" Board not detected at pose {i+1}, skip.") continue corners = cv2.cornerSubPix( gray, corners, (11, 11), (-1, -1), (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)) ret, rvec, tvec = cv2.solvePnP( self.objp, corners, self.K, self.dist) R, _ = cv2.Rodrigues(rvec) R_target2cam.append(R) t_target2cam.append(tvec.flatten()) return (R_gripper2base, t_gripper2base, R_target2cam, t_target2cam) def calibrate_eye_in_hand(self, R_g2b, t_g2b, R_t2c, t_t2c): """Eye-in-hand: camera mounted on end-effector. Solves for T_camera_to_gripper.""" R, t = cv2.calibrateHandEye( R_g2b, t_g2b, R_t2c, t_t2c, method=cv2.CALIB_HAND_EYE_TSAI # Also: PARK, HORAUD, DANIILIDIS ) T_cam2gripper = np.eye(4) T_cam2gripper[:3, :3] = R T_cam2gripper[:3, 3] = t.flatten() return T_cam2gripper def calibrate_eye_to_hand(self, R_g2b, t_g2b, R_t2c, t_t2c): """Eye-to-hand: camera fixed in workspace. Solves for T_camera_to_base.""" # Invert robot poses (base-to-gripper → gripper-to-base) R_b2g = [R.T for R in R_g2b] t_b2g = [-R.T @ t for R, t in zip(R_g2b, t_g2b)] R, t = cv2.calibrateHandEye( R_b2g, t_b2g, R_t2c, t_t2c, method=cv2.CALIB_HAND_EYE_TSAI ) T_cam2base = np.eye(4) T_cam2base[:3, :3] = R T_cam2base[:3, 3] = t.flatten() return T_cam2base def verify_calibration(self, T_cam2ee, robot, camera, target_points_3d): """Verify by projecting a known 3D point through the full chain. Error should be Optional[StampedFrame]: """Get most recent frame (non-blocking). Returns None if empty.""" with self._lock: if self._buffer: return self._buffer[-1] return None def wait_for_frame(self, timeout=1.0) -> Optional[StampedFrame]: """Block until a new frame arrives""" self._new_frame.clear() if self._new_frame.wait(timeout=timeout): return self.get_latest() return None def get_diagnostics(self) -> dict: """Streaming health metrics""" if self._capture_times: times = list(self._capture_times) fps = 1.0 / np.mean(times) if np.mean(times) > 0 else 0 else: fps = 0 return { "sensor": self.name, "fps": round(fps, 1), "frames_captured": self._sequence, "frames_dropped": self._drop_count, "buffer_size": len(self._buffer), "avg_capture_ms": round(np.mean(times) * 1000, 1) if self._capture_times else 0, } ``` ### Multi-Sensor Synchronization ```python class SyncedMultiSensor: """Synchronize frames from multiple sensors by timestamp. Uses nearest-neighbor matching within a time tolerance. For hardware-synced sensors, use hardware trigger instead. """ def __init__(self, sensors: dict, max_time_diff_ms=33): """ Args: sensors: {"rgb": CameraStream, "depth": CameraStream, ...} max_time_diff_ms: Maximum allowed time difference between synced frames. Default 33ms (1 frame at 30Hz). """ self.sensors = sensors self.max_dt = max_time_diff_ms / 1000.0 self._synced_callback = None def start(self): for s in self.sensors.values(): s.start() def stop(self): for s in self.sensors.values(): s.stop() def get_synced(self) -> Optional[dict]: """Get time-synchronized frames from all sensors. Returns None if any sensor is missing or too far out of sync.""" frames = {} for name, stream in self.sensors.items(): frame = stream.get_latest() if frame is None: return None frames[name] = frame # Check time alignment against the first sensor ref_time = list(frames.values())[0].timestamp for name, frame in frames.items(): dt = abs(frame.timestamp - ref_time) if dt > self.max_dt: return None # Out of sync return frames def get_synced_interpolated(self) -> Optional[dict]: """For sensors at different rates, interpolate to common timestamp. Useful for IMU + camera fusion.""" # Get latest from each sensor frames = {} for name, stream in self.sensors.items(): frame = stream.get_latest() if frame is None: return None frames[name] = frame # Use the SLOWEST sensor's timestamp as reference ref_time = min(f.timestamp for f in frames.values()) result = {} for name, frame in frames.items(): dt = frame.timestamp - ref_time if abs(dt) list: """Run detection with robotics post-processing""" # Raw detection raw_dets = self.model(rgb) # Filter by confidence dets = [d for d in raw_dets if d.confidence >= self.min_confidence] # Estimate 3D position if depth is available if depth is not None: for det in dets: det.position_3d = self._backproject(det.center, depth) # Filter by workspace if self.workspace_bounds and det.position_3d is not None: if not self.workspace_bounds.contains(det.position_3d): det.in_workspace = False continue det.in_workspace = True # Track across frames for stability tracked = self.tracker.update(dets) return tracked def _backproject(self, pixel, depth_image): """Convert 2D pixel + depth to 3D point in camera frame. This is the INVERSE … ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [arpitg1304](https://github.com/arpitg1304) - **Source:** [arpitg1304/robotics-agent-skills](https://github.com/arpitg1304/robotics-agent-skills) - **License:** Apache-2.0 Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** yes - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-arpitg1304-robotics-agent-skills-robot-perception - Seller: https://agentstack.voostack.com/s/arpitg1304 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.