Mediapipe Tracking

MediaPipe Tracking

When to Use This Skill

• Tracking human body pose in real-time video
• Detecting and tracking hand gestures or finger positions
• Building face filters, expression analysis, or gaze tracking
• Needing 3D depth information (Z-axis) for body landmarks
• Creating fitness apps, sign language recognition, or virtual avatars

Decision Framework

Which MediaPipe Solution?

What body part do you need to track?
├── Full body pose (standing, sitting, exercising)
│   └── ✅ Pose Estimation — 33 landmarks, 3D (x, y, z)
│
├── Hands and fingers
│   └── ✅ Hand Tracking — 21 landmarks per hand
│
├── Face details (lips, eyes, eyebrows, jawline)
│   └── ✅ Face Mesh — 468 landmarks, 3D
│
└── Multiple tasks simultaneously
    └── ✅ Holistic — combines Pose + Hands + Face

MediaPipe vs YOLO Pose Estimation

| Feature | MediaPipe Pose | YOLO Pose | |---------|---------------|----------| | Landmarks | 33 (full body + face) | 17 (body only) | | Depth (Z-axis) | ✅ Yes (camera-relative) | ❌ No (2D only) | | Speed | ⚡ Very fast (optimized for mobile) | ⚡ Fast | | Multi-person | ❌ Single person per frame | ✅ Multiple people | | 3D Avatar | ✅ Possible (Z depth) | ❌ Not possible | | Hand details | ✅ 21 points per hand (separate solution) | ❌ Not available | | Face details | ✅ 468 points (separate solution) | ❌ Not available | | Best for | Single-user apps, gesture control, fitness | Multi-person detection, surveillance |

> Rule of thumb: Need detailed single-person anatomy (hands, face, depth)? → MediaPipe. Need to detect multiple people quickly? → YOLO Pose.

Landmark Counts Reference

| Solution | Points | Notable Landmarks | |----------|--------|--------------------| | Pose | 33 | Nose, shoulders, elbows, wrists, hips, knees, ankles, eyes, ears | | Hand | 21 | Wrist, thumb (4 joints), index (4), middle (4), ring (4), pinky (4) | | Face Mesh | 468 | Lips contour, eye contour, eyebrows, nose bridge, jawline, iris |

Integration Guide

Pose Estimation Setup

import cv2
import mediapipe as mp

mp_pose = mp.solutions.pose
mp_draw = mp.solutions.drawing_utils

cap = cv2.VideoCapture(0)
with mp_pose.Pose(min_detection_confidence=0.5,
                   min_tracking_confidence=0.5) as pose:
    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        
        # MediaPipe expects RGB
        rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        results = pose.process(rgb)
        
        if results.pose_landmarks:
            mp_draw.draw_landmarks(
                frame, results.pose_landmarks,
                mp_pose.POSE_CONNECTIONS)
        
        cv2.imshow('Pose', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

cap.release()
cv2.destroyAllWindows()

Hand Tracking Setup

mp_hands = mp.solutions.hands

with mp_hands.Hands(max_num_hands=2,
                     min_detection_confidence=0.7) as hands:
    # Same loop pattern as pose
    results = hands.process(rgb)
    if results.multi_hand_landmarks:
        for hand_landmarks in results.multi_hand_landmarks:
            mp_draw.draw_landmarks(
                frame, hand_landmarks,
                mp_hands.HAND_CONNECTIONS)

Face Mesh Setup

mp_face = mp.solutions.face_mesh

with mp_face.FaceMesh(max_num_faces=1,
                       min_detection_confidence=0.5) as face_mesh:
    results = face_mesh.process(rgb)
    if results.multi_face_landmarks:
        for face_landmarks in results.multi_face_landmarks:
            mp_draw.draw_landmarks(
                frame, face_landmarks,
                mp_face.FACEMESH_TESSELATION)

Critical Gotchas

1. MediaPipe Expects RGB, Not BGR

MediaPipe processes RGB images. OpenCV captures BGR. Always convert:

rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
results = pose.process(rgb)

2. Landmark Coordinates Are Normalized (0-1)

Landmark .x and .y are normalized to image dimensions. To get pixel coordinates:

h, w, _ = frame.shape
lm = results.pose_landmarks.landmark[mp_pose.PoseLandmark.LEFT_SHOULDER]
pixel_x, pixel_y = int(lm.x * w), int(lm.y * h)

3. Z-Depth Is Relative, Not Absolute

The .z coordinate represents depth relative to the hip center (Pose) or wrist (Hand). It is NOT world-scale distance in meters.

4. Single Person Limitation

MediaPipe Pose processes one person per frame. For multi-person scenarios, use YOLO Pose or run MediaPipe per detected person crop.

5. Use the Context Manager

Always use with statement for proper resource cleanup:

# CORRECT
with mp_pose.Pose() as pose:
    ...

# AVOID
pose = mp_pose.Pose()
# resources may leak

Application Ideas

| Application | Solution | Key Landmarks | |-------------|----------|----------------| | Exercise rep counter | Pose | Shoulder, elbow, wrist angles | | Sign language recognition | Hand | All 21 finger joints | | Drowsiness detection | Face Mesh | Eye aspect ratio (EAR) | | Virtual try-on | Pose + Face | Body proportions + face alignment | | Gesture-controlled UI | Hand | Fingertip positions (index, thumb) | | Scoliosis screening | Pose | Shoulder/hip symmetry analysis |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aeren23
• Source: aeren23/image-processing-skills
• License: MIT

Install and usage instructions live in the source repository linked above.