Forge Edge Detection

Edge Detection

Add post-process edge detection, stencil X-ray vision, and Fresnel ghost silhouettes to an SDL GPU project. Uses G-buffer MRT output (depth + normals) for Sobel edge detection, depth_fail_op for occluded-object marking, and additive blending for translucent ghost rendering.

Based on [GPU Lesson 36](../../../lessons/gpu/36-edge-detection/).

When to use

• You want post-process outlines from depth/normal discontinuities
• You need X-ray vision to reveal occluded objects behind walls
• You want ghostly translucent silhouettes for hidden geometry
• You need an alternative to L34's stencil-expansion outlines

Pipeline overview (6 pipelines)

1. Shadow pipeline — Depth-only pass for shadow mapping
2. Scene pipeline (MRT) — Renders lit scene to two color targets:

color (RT0) and view-space normals (RT1), plus depth-stencil

1. Grid pipeline (MRT) — Same MRT layout for the floor grid
2. Edge composite pipeline — Full-screen quad, samples depth + normal

textures, runs Sobel filter, composites edges onto the scene

1. X-ray mark pipeline — Draws occluded objects with `depthfailop =

INCREMENTANDWRAP, colorwritemask = 0`, writing only to stencil where fragments fail the depth test

1. Ghost pipeline — Draws the same objects again with stencil test

NOT_EQUAL ref=0 and additive blending for Fresnel ghost silhouettes

G-buffer setup (MRT)

The scene pass renders to two color targets simultaneously:

/* RT0: scene color (RGBA8) */
/* RT1: view-space normals (RGBA16_FLOAT or RGBA8_SNORM) */
SDL_GPUColorTargetInfo color_targets[2] = {
    { .texture = scene_color_tex, .load_op = SDL_GPU_LOADOP_CLEAR },
    { .texture = normal_tex,      .load_op = SDL_GPU_LOADOP_CLEAR },
};

The fragment shader outputs to both targets:

struct PSOutput {
    float4 color  : SV_Target0;
    float4 normal : SV_Target1;
};

Pipeline must declare two color target descriptions to match.

Sobel edge detection

Sample depth and normal textures in a 3x3 neighborhood. Apply Sobel operators (horizontal Gx and vertical Gy kernels) separately to depth and to each normal component. Combine magnitudes:

/* Sobel kernels */
/* Gx = [-1 0 1; -2 0 2; -1 0 1] */
/* Gy = [-1 -2 -1; 0 0 0; 1 2 1] */

float depth_edge = sqrt(gx_depth * gx_depth + gy_depth * gy_depth);
float normal_edge = sqrt(gx_nx*gx_nx + gy_nx*gy_nx
                       + gx_ny*gx_ny + gy_ny*gy_ny
                       + gx_nz*gx_nz + gy_nz*gy_nz);

float edge = saturate(depth_edge * depth_scale + normal_edge * normal_scale);

Threshold and composite: final_color = lerp(scene_color, edge_color, edge).

X-ray mark technique (depthfailop)

The key insight: depth_fail_op fires when a fragment is behind existing geometry. By drawing the target object with this operation, we mark exactly the pixels where it is occluded.

/* X-ray mark pipeline stencil state */
SDL_GPUStencilOpState xray_stencil = {
    .fail_op       = SDL_GPU_STENCILOP_KEEP,
    .pass_op       = SDL_GPU_STENCILOP_KEEP,           /* visible parts: no mark */
    .depth_fail_op = SDL_GPU_STENCILOP_INCREMENT_AND_WRAP, /* occluded: mark */
    .compare_op    = SDL_GPU_COMPAREOP_ALWAYS,
};

Color write mask = 0 and depth write = false. This pass is invisible — it only writes stencil values where the object is hidden behind other geometry.

Ghost silhouette technique

After the X-ray mark pass, draw the object again with:

• Stencil test: compare_op = NOT_EQUAL, reference = 0 (draw only where marked)
• Additive blending: src_color_blendfactor = ONE, dst = ONE
• Fresnel-based opacity: brighter at edges where dot(N, V) is low

/* Fresnel ghost effect */
float fresnel = 1.0 - saturate(dot(normal, view_dir));
fresnel = pow(fresnel, fresnel_power);
return float4(ghost_color * fresnel * intensity, 0.0);

The additive blend makes the ghost glow through geometry without needing alpha sorting.

Clearing stencil

Clear stencil alongside depth at the start of the render pass:

SDL_GPUDepthStencilTargetInfo ds_target = {
    .texture          = depth_stencil_texture,
    .load_op          = SDL_GPU_LOADOP_CLEAR,
    .store_op         = SDL_GPU_STOREOP_STORE,
    .clear_depth      = 1.0f,
    .clear_stencil    = 0,
    .stencil_load_op  = SDL_GPU_LOADOP_CLEAR,
    .stencil_store_op = SDL_GPU_STOREOP_STORE,
};

Key patterns

| Pattern | Value | |---------|-------| | depth_fail_op | SDL_GPU_STENCILOP_INCREMENT_AND_WRAP | | X-ray mark color_write_mask | 0 (invisible pass) | | X-ray mark depth_write | false | | Ghost stencil compare_op | SDL_GPU_COMPAREOP_NOT_EQUAL | | Ghost stencil reference | 0 | | Ghost blend | Additive (SRC=ONE, DST=ONE) | | MRT count | 2 (color + normals) |

Key API calls

• SDL_BeginGPURenderPass(cmd, color_targets, 2, &ds) — MRT with 2 color targets
• SDL_SetGPUStencilReference(pass, ref) — set stencil compare reference value
• SDL_DrawGPUPrimitives(pass, 3, 1, 0, 0) — fullscreen triangle (no vertex buffer)
• SDL_GPU_STENCILOP_INCREMENT_AND_WRAP — depthfailop for X-ray marking
• SDL_GPU_COMPAREOP_NOT_EQUAL — stencil test for ghost pass (ref=0, pass where >0)
• SDL_GPU_BLENDFACTOR_ONE (src + dst) — additive blending for ghost glow
• SDL_GPUTextureSupportsFormat — negotiate D24S8 with SAMPLER usage

Common mistakes

• Forgetting MRT pipeline color target count — The pipeline must declare

the same number of color target descriptions as render pass color targets. Mismatched counts cause validation errors or undefined behavior.

• Wrong depthfailop direction — depth_fail_op fires when the fragment

is behind existing depth. Make sure the scene depth is already established before the X-ray mark pass.

• Not disabling depth writes for X-ray mark — If the mark pass writes

depth, it corrupts the depth buffer and prevents the ghost pass from rendering correctly.

• Missing stencil clear — Stale stencil values from the previous frame

cause ghost artifacts everywhere. Always clear stencil at pass start.

• Sobel on linear depth vs hyperbolic depth — Raw depth buffer values

are hyperbolic (non-linear). For better edge detection, linearize depth first or adjust the depth scale threshold.

Cross-references

• [GPU Lesson 36 — Edge Detection](../../../lessons/gpu/36-edge-detection/)

for the full walkthrough

• [GPU Lesson 34 — Stencil Testing](../../../lessons/gpu/34-stencil-testing/)

for stencil buffer fundamentals and stencil-expansion outlines

• [GPU Lesson 27 — SSAO](../../../lessons/gpu/27-ssao/)

for G-buffer and view-space normal techniques

• [GPU Lesson 15 — Cascaded Shadow Maps](../../../lessons/gpu/15-cascaded-shadow-maps/)

for depth-only render passes

• [GPU Lesson 16 — Blending](../../../lessons/gpu/16-blending/)

for blend state configuration (additive blending)

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Nebulavenus
• Source: Nebulavenus/forge-gpu
• License: Zlib

Install and usage instructions live in the source repository linked above.