Automotive Ai Ecu

Automotive Ai Ecu

Camera Vision Ai

Camera Vision AI for Automotive

Skill: Computer vision pipelines for automotive cameras with AI/ML integration Version: 1.0.0 Category: AI-ECU / Perception Complexity: Advanced

---

Overview

Complete guide to automotive camera vision AI pipelines: object detection (YOLO, EfficientDet), semantic segmentation, lane detection, 360° surround view with AI, camera ISP tuning, and multi-camera fusion for ADAS and autonomous driving.

Automotive Camera Landscape

Camera Types in Modern Vehicles

| Camera Type | Resolution | FOV | Frame Rate | Use Case | Interface | |-------------|------------|-----|------------|----------|-----------| | Front Camera | 1920x1080 - 2880x1644 | 60-120° | 30-60 FPS | ADAS, Lane Keep, AEB | MIPI CSI-2 | | Rear Camera | 1280x720 - 1920x1080 | 120-180° | 30 FPS | Parking, Rear Cross Traffic | MIPI CSI-2 | | Side Cameras (2x) | 1280x720 | 90-120° | 30 FPS | Blind Spot, Lane Change | MIPI CSI-2 | | DMS Camera (IR) | 640x480 - 1280x720 | 60-90° | 30-60 FPS | Driver Monitoring | MIPI CSI-2 | | OMS Camera (IR) | 640x480 | 90-120° | 15-30 FPS | Occupant Monitoring | MIPI CSI-2 | | 360° Surround | 4x 1280x720 | 180-220° | 30 FPS | Parking, Top View | MIPI CSI-2 |

Total Bandwidth: Up to 12 Gbps for multi-camera system (8 cameras)

---

Camera ISP Pipeline

Image Signal Processor (ISP) Tuning

ISP Pipeline: Raw Bayer → Demosaic → White Balance → Gamma → Color Correction → AI Inference

class AutomotiveISPTuner:
    """
    ISP tuning for automotive vision AI
    Goal: Optimize image quality for ML model accuracy (not human perception)
    """
    def __init__(self, isp_device):
        self.isp = isp_device

    def tune_for_object_detection(self):
        """
        ISP tuning optimized for YOLO/EfficientDet
        - High contrast for edge detection
        - Low noise to avoid false positives
        - Wide dynamic range (HDR) for varying light conditions
        """
        self.isp.set_parameter('demosaic_algorithm', 'bilinear')  # Fast, good for edges
        self.isp.set_parameter('white_balance_mode', 'auto')  # Auto WB for varying conditions
        self.isp.set_parameter('gamma', 2.2)  # Standard gamma
        self.isp.set_parameter('contrast', 1.3)  # +30% contrast for better edges
        self.isp.set_parameter('sharpening', 1.5)  # +50% sharpening
        self.isp.set_parameter('noise_reduction', 'moderate')  # Balance speed vs. quality
        self.isp.set_parameter('hdr_mode', 'enabled')  # HDR for tunnels, bright sun
        self.isp.set_parameter('ae_target', 0.5)  # Exposure target (0-1 scale)

    def tune_for_lane_detection(self):
        """
        ISP tuning for lane marking detection
        - High contrast for white/yellow lines on asphalt
        - Aggressive edge enhancement
        - No color correction (monochrome sufficient)
        """
        self.isp.set_parameter('contrast', 1.5)  # +50% contrast
        self.isp.set_parameter('sharpening', 2.0)  # Maximum sharpening
        self.isp.set_parameter('saturation', 0.8)  # Reduce saturation (focus on luminance)
        self.isp.set_parameter('edge_enhancement', 'aggressive')

    def tune_for_dms(self):
        """
        ISP tuning for IR-based driver monitoring
        - 940nm IR illumination
        - No color processing (monochrome sensor)
        - Low noise for accurate eye/face detection
        """
        self.isp.set_parameter('ir_filter', 'bypass')  # Allow 940nm IR
        self.isp.set_parameter('noise_reduction', 'aggressive')  # Critical for DMS accuracy
        self.isp.set_parameter('gain', 2.0)  # Amplify IR signal
        self.isp.set_parameter('frame_rate', 60)  # High FPS for gaze tracking

    def adaptive_tuning_based_on_scenario(self, scenario):
        """
        Dynamically adjust ISP based on driving scenario
        """
        if scenario == 'highway_day':
            self.isp.set_parameter('exposure_time', 8)  # ms (bright conditions)
            self.isp.set_parameter('gain', 1.0)

        elif scenario == 'highway_night':
            self.isp.set_parameter('exposure_time', 20)  # ms (low light)
            self.isp.set_parameter('gain', 4.0)  # Amplify signal
            self.isp.set_parameter('noise_reduction', 'aggressive')

        elif scenario == 'tunnel_entry':
            self.isp.set_parameter('hdr_mode', 'enabled')  # Critical for tunnel transitions
            self.isp.set_parameter('ae_speed', 'fast')  # Quickly adapt to light change

        elif scenario == 'parking':
            self.isp.set_parameter('fisheye_correction', 'enabled')  # Correct distortion
            self.isp.set_parameter('frame_rate', 30)  # Standard FPS sufficient

# Example: Apply ISP tuning
isp = AutomotiveISPTuner('/dev/video0')
isp.tune_for_object_detection()

---

Object Detection Pipelines

YOLOv5 for Automotive ADAS

Use Case: Real-time multi-class object detection (vehicles, pedestrians, cyclists, traffic signs)

import cv2
import numpy as np
import torch

class AutomotiveYOLOv5:
    """
    YOLOv5 optimized for automotive ADAS
    Classes: vehicle, pedestrian, cyclist, motorcycle, bus, truck, traffic_light, stop_sign
    """
    def __init__(self, model_path, npu_runtime='snpe', confidence_threshold=0.5):
        self.model = self.load_model(model_path, npu_runtime)
        self.conf_thresh = confidence_threshold
        self.iou_thresh = 0.45
        self.classes = ['vehicle', 'pedestrian', 'cyclist', 'motorcycle',
                       'bus', 'truck', 'traffic_light', 'stop_sign']

    def load_model(self, model_path, runtime):
        """Load quantized YOLOv5s on NPU"""
        if runtime == 'snpe':
            import snpe
            container = snpe.load_container(model_path)
            network = snpe.build_network(container, snpe.SNPE_Runtime.RUNTIME_HTA)
            return network
        elif runtime == 'tflite':
            import tflite_runtime.interpreter as tflite
            interpreter = tflite.Interpreter(
                model_path=model_path,
                experimental_delegates=[tflite.load_delegate('libvx_delegate.so')]
            )
            interpreter.allocate_tensors()
            return interpreter

    def preprocess(self, frame):
        """Preprocess frame for YOLO inference"""
        # Resize to 640x640 (YOLOv5 input)
        resized = cv2.resize(frame, (640, 640))

        # Normalize to [0, 1]
        normalized = resized.astype(np.float32) / 255.0

        # HWC → CHW (Height, Width, Channels → Channels, Height, Width)
        transposed = np.transpose(normalized, (2, 0, 1))

        # Add batch dimension
        batched = np.expand_dims(transposed, axis=0)

        return batched

    def infer(self, frame):
        """Run inference on frame"""
        preprocessed = self.preprocess(frame)

        # Run on NPU
        output = self.model.execute({'images': preprocessed})

        # Postprocess YOLO output
        detections = self.postprocess(output['output'], frame.shape)

        return detections

    def postprocess(self, output, original_shape):
        """
        Postprocess YOLO output
        Output shape: [1, 25200, 13] (25200 anchors, 13 = 4 bbox + 1 conf + 8 classes)
        """
        output = output[0]  # Remove batch dimension

        # Extract bounding boxes, confidence, class scores
        boxes = output[:, :4]  # [x, y, w, h]
        confidences = output[:, 4]  # objectness score
        class_scores = output[:, 5:]  # class probabilities

        # Filter by confidence threshold
        mask = confidences > self.conf_thresh
        boxes = boxes[mask]
        confidences = confidences[mask]
        class_scores = class_scores[mask]

        # Get class predictions
        class_ids = np.argmax(class_scores, axis=1)
        class_confidences = np.max(class_scores, axis=1)

        # Final confidence = objectness * class_confidence
        final_confidences = confidences * class_confidences

        # Non-Maximum Suppression (NMS)
        indices = self.nms(boxes, final_confidences, self.iou_thresh)

        # Build detection results
        detections = []
        for i in indices:
            x, y, w, h = boxes[i]

            # Convert from YOLO format (center_x, center_y, width, height) to (x1, y1, x2, y2)
            x1 = int((x - w/2) * original_shape[1] / 640)
            y1 = int((y - h/2) * original_shape[0] / 640)
            x2 = int((x + w/2) * original_shape[1] / 640)
            y2 = int((y + h/2) * original_shape[0] / 640)

            detections.append({
                'class': self.classes[class_ids[i]],
                'class_id': int(class_ids[i]),
                'confidence': float(final_confidences[i]),
                'bbox': [x1, y1, x2, y2]
            })

        return detections

    def nms(self, boxes, scores, iou_threshold):
        """Non-Maximum Suppression"""
        x1 = boxes[:, 0] - boxes[:, 2] / 2
        y1 = boxes[:, 1] - boxes[:, 3] / 2
        x2 = boxes[:, 0] + boxes[:, 2] / 2
        y2 = boxes[:, 1] + boxes[:, 3] / 2

        areas = (x2 - x1) * (y2 - y1)
        order = scores.argsort()[::-1]

        keep = []
        while order.size > 0:
            i = order[0]
            keep.append(i)

            # Compute IoU of kept box with all remaining boxes
            xx1 = np.maximum(x1[i], x1[order[1:]])
            yy1 = np.maximum(y1[i], y1[order[1:]])
            xx2 = np.minimum(x2[i], x2[order[1:]])
            yy2 = np.minimum(y2[i], y2[order[1:]])

            w = np.maximum(0.0, xx2 - xx1)
            h = np.maximum(0.0, yy2 - yy1)
            inter = w * h

            iou = inter / (areas[i] + areas[order[1:]] - inter)

            # Keep boxes with IoU below threshold
            inds = np.where(iou  0.5:
                y1, x1, y2, x2 = boxes[i]
                detections.append({
                    'class_id': int(classes[i]),
                    'confidence': float(scores[i]),
                    'bbox': [
                        int(x1 * frame.shape[1]),
                        int(y1 * frame.shape[0]),
                        int(x2 * frame.shape[1]),
                        int(y2 * frame.shape[0])
                    ]
                })

        return detections

# Benchmark comparison:
# YOLOv5s: 18ms latency, 37.4 mAP @ COCO
# EfficientDet-D0: 42ms latency, 33.8 mAP @ COCO
# EfficientDet-D2: 75ms latency, 43.0 mAP @ COCO

---

Semantic Segmentation

Lane Detection with LaneNet

Use Case: Pixel-level lane marking detection for lane keeping assist (LKA)

class LaneNetSegmentation:
    """
    LaneNet for pixel-level lane detection
    Output: Binary segmentation mask (lane pixels vs. background)
    """
    def __init__(self, model_path):
        self.model = self.load_model(model_path)
        self.input_size = (512, 256)  # Width x Height

    def load_model(self, model_path):
        """Load LaneNet model"""
        import snpe
        container = snpe.load_container(model_path)
        return snpe.build_network(container, snpe.SNPE_Runtime.RUNTIME_HTA)

    def preprocess(self, frame):
        """Preprocess frame for LaneNet"""
        # Crop bottom half of frame (road region)
        height = frame.shape[0]
        cropped = frame[height//2:, :]

        # Resize to input size
        resized = cv2.resize(cropped, self.input_size)

        # Normalize
        normalized = resized.astype(np.float32) / 255.0

        # CHW format
        transposed = np.transpose(normalized, (2, 0, 1))
        batched = np.expand_dims(transposed, axis=0)

        return batched, height//2

    def infer(self, frame):
        """Run LaneNet inference"""
        preprocessed, crop_offset = self.preprocess(frame)

        # Run inference
        output = self.model.execute({'input': preprocessed})

        # Output: [1, 2, 256, 512] (binary segmentation: lane vs. background)
        segmentation = output['segmentation'][0]

        # Get lane mask (class 1)
        lane_mask = segmentation[1]  # [256, 512]

        # Resize back to original size
        lane_mask_resized = cv2.resize(lane_mask, (frame.shape[1], frame.shape[0]//2))

        # Create full-size mask
        full_mask = np.zeros((frame.shape[0], frame.shape[1]), dtype=np.float32)
        full_mask[crop_offset:, :] = lane_mask_resized

        return full_mask

    def extract_lane_lines(self, lane_mask, threshold=0.5):
        """
        Extract polynomial lane lines from segmentation mask
        Fit 2nd order polynomial: y = ax^2 + bx + c
        """
        # Threshold mask
        binary_mask = (lane_mask > threshold).astype(np.uint8) * 255

        # Find lane pixels
        lane_pixels = np.where(binary_mask > 0)
        y_pixels = lane_pixels[0]
        x_pixels = lane_pixels[1]

        if len(x_pixels)  0:
            x, y, w, h = faces[0]
            face_region = frame[y:y+h, x:x+w]

            # Calculate mean brightness
            mean_brightness = np.mean(face_region)

            # Target brightness: 128 (50% of 255)
            target_brightness = 128
            error = target_brightness - mean_brightness

            # Proportional control
            intensity_adjust = error * 0.5  # Proportional gain
            current_intensity = self.ir_pwm.ChangeDutyCycle
            new_intensity = np.clip(current_intensity + intensity_adjust, 10, 100)

            self.set_ir_intensity(new_intensity)

    def capture_frame(self):
        """Capture IR frame with auto-adjustment"""
        ret, frame = self.cap.read()
        if ret:
            # Convert to grayscale (IR is already monochrome)
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            # Auto-adjust IR intensity
            self.auto_adjust_ir(gray)

            return gray
        return None

# Usage
dms_camera = DMSCameraController()
frame = dms_camera.capture_frame()

---

Face and Eye Detection

MediaPipe Face Mesh for Landmark Detection

Face Landmarks: 468 3D points covering face geometry (eyes, nose, mouth, contours)

import mediapipe as mp

class FaceLandmarkDetector:
    """
    Detect 468 face landmarks using MediaPipe
    Optimized for automotive DMS (lightweight model on NPU)
    """
    def __init__(self):
        self.mp_face_mesh = mp.solutions.face_mesh
        self.face_mesh = self.mp_face_mesh.FaceMesh(
            static_image_mode=False,
            max_num_faces=1,
            refine_landmarks=True,  # Enable iris landmarks
            min_detection_confidence=0.5,
            min_tracking_confidence=0.5
        )

        # Key landmark indices
        self.LEFT_EYE_INDICES = [33, 133, 160, 159, 158, 144, 145, 153]
        self.RIGHT_EYE_INDICES = [362, 263, 387, 386, 385, 373, 374, 380]
        self.LEFT_IRIS_INDICES = [468, 469, 470, 471, 472]
        self.RIGHT_IRIS_INDICES = [473, 474, 475, 476, 477]

    def detect(self, frame):
        """
        Detect face landmarks in IR frame
        Returns: 468 (x, y, z) landmarks in normalized coordinates [0, 1]
        """
        # Convert grayscale to RGB (MediaPipe expects RGB)
        frame_rgb = cv2.cvtColor(frame, cv2.COLOR_GRAY2RGB)

        # Run MediaPipe
        results = self.face_mesh.process(frame_rgb)

        if results.multi_face_landmarks:
            landmarks = results.multi_face_landmarks[0]

            # Convert to numpy array
            h, w = frame.shape[:2]
            landmarks_array = np.array([
                [lm.x * w, lm.y * h, lm.z * w]  # Denormalize to pixel coordinates
                for lm in landmarks.landmark
            ])

            return landmarks_array
        return None

    def get_eye_landmarks(self, landmarks):
        """Extract left and right eye landmarks"""
        if landmarks is None:
            return None, None

        left_eye = landmarks[self.LEFT_EYE_INDICES]
        right_eye = landma

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [pangzhenying2025](https://github.com/pangzhenying2025)
- **Source:** [pangzhenying2025/hermes-automotive-skills](https://github.com/pangzhenying2025/hermes-automotive-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.