Neural Reality Capture

Neural Reality Capture

Neural reality capture is the production craft of reconstructing or rendering a real subject from many overlapping observations. In this skill, the deliverable is one of three families:

• a photogrammetric mesh with textures, usually for game, VFX, GIS, cultural heritage, ecommerce, or DCC handoff;
• a NeRF or related neural radiance field, usually for high-quality novel-view rendering from known camera poses;
• a 3D Gaussian splat scene, usually for real-time free-viewpoint playback with radiance-field appearance.

Stay provider-independent. Choose tools by evidence, constraints, and handoff needs, not by brand habit. Do not promise metrology unless the request includes surveyed control, calibrated capture, uncertainty reporting, and a qualified measurement workflow. Do not expand into text-to-3D, generative resculpting, animation rigging, material lookdev beyond captured handoff, or broad finishing work that belongs downstream.

Evidence Labels

Use these labels in plans, reviews, and troubleshooting:

• Paper fact: a claim from a peer-reviewed paper or original technical report, such as the original NeRF or 3D Gaussian Splatting papers.
• Standard fact: a claim from a specification or standard, such as Khronos glTF or OpenUSD documentation.
• Official documentation fact: a claim from tool, capture, or production documentation maintained by the relevant organization.
• Empirical observation: a result from a documented test or reconstruction run.
• Production heuristic: a practical rule that usually improves capture or handoff, but is not a guarantee.

When the user asks for a recommendation, combine fact and heuristic explicitly: "Standard fact: glTF 2.0 uses meters and a right-handed coordinate system. Production heuristic: deliver GLB for lightweight web review and USD for layered VFX or DCC assembly."

Representation Choice

Choose the representation from the final use, not from capture novelty.

Photogrammetric mesh

Use a mesh when the asset must be edited, decimated, baked, collision-tested, physically placed, measured with documented caveats, 3D printed, or delivered to engines and DCC tools as conventional geometry.

Strengths:

• explicit topology, UVs, textures, normals, and material channels;
• established interchange through OBJ, FBX, Alembic, glTF/GLB, and USD;
• compatible with cleanup, retopology, collision, LOD, baking, and engine import;
• can be scaled and georeferenced with markers, scale bars, GCPs, or surveyed control.

Risks:

• reflective, transparent, hair-like, thin, repetitive, or textureless surfaces reconstruct poorly;
• topology and UVs are often production-unfriendly until cleaned;
• baked textures can hide geometric defects;
• high accuracy requires controlled capture, calibration, control points, and QA, not only many photos.

NeRF

Use a NeRF when photorealistic novel-view rendering matters more than mesh editability, especially for bounded inward-facing captures, view-dependent appearance, transparency-like effects, and difficult glossy appearance that a quick mesh bake would mishandle.

Paper fact: the original NeRF paper, Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020), represents a scene with a continuous 5D function of spatial location and viewing direction that outputs volume density and view-dependent radiance. It optimizes from input images with known camera poses and renders by differentiable volume rendering along camera rays. Verified 2026-07-11 from the authors' project page and paper.

Strengths:

• strong novel-view appearance for captured radiance;
• handles view-dependent effects better than many naive texture bakes;
• useful for virtual tours, hero renders, volumetric previews, and reference playback.

Risks:

• camera poses and image consistency still matter;
• dynamic objects, exposure shifts, rolling shutter, motion blur, and focus changes become learned artifacts;
• mesh extraction is possible in some workflows but is not the primary representation and may be noisy;
• delivery is less standardized than mesh delivery, and runtime support depends on the target.

3D Gaussian splat

Use a Gaussian splat when real-time free-viewpoint playback of a captured scene is the priority and the target viewer supports splats. It is often a strong choice for rooms, streets, cultural sites, products with complex appearance, and location previews where interactive viewing beats editable topology.

Paper fact: the 2023 SIGGRAPH paper 3D Gaussian Splatting for Real-Time Radiance Field Rendering starts from sparse points produced during camera calibration, represents scenes with optimized 3D Gaussians, includes interleaved optimization and density control, and uses a visibility-aware splatting renderer to achieve high-quality real-time rendering in their evaluated conditions. Verified 2026-07-11 from the authors' Inria project page and paper.

Strengths:

• fast interactive viewing compared with many neural-field renderers;
• strong appearance preservation for complex captured scenes;
• works well when the final deliverable is an immersive viewer rather than editable geometry.

Risks:

• splats are not conventional solid meshes;
• collision, CAD, 3D printing, and DCC editing require conversion or a separate mesh;
• file formats, compression, and engine support are still volatile compared with glTF mesh delivery;
• floaters and semi-transparent halos can be severe around thin, shiny, moving, or under-observed content.

Quick decision table

| Constraint | Preferred representation | Reason | |---|---|---| | Game prop needs collision, retopo, LODs | Mesh | Editable geometry and standard engine pipeline | | Museum room needs browser-like free-view walkthrough | 3DGS or NeRF | View synthesis preserves captured appearance | | Architectural survey with GCPs and dimensional checks | Mesh plus survey records | Conventional geometry and QA artifacts are inspectable | | Reflective retail object for turntable preview | Try NeRF/3DGS, plus controlled mesh capture if needed | View-dependent appearance may beat a quick texture bake | | Asset must ship as GLB to ecommerce | Mesh | glTF/GLB is runtime asset delivery, not a NeRF container | | VFX layout wants layered scene assembly | Mesh or point/splat reference in USD package | USD is strong for composition, payloads, variants, and asset references |

Capture Planning

The capture plan is the cheapest place to prevent reconstruction failure. Start with the target use, subject type, rights, schedule, weather or lighting, scale strategy, and the required deliverable.

Universal capture requirements

Official documentation fact: COLMAP is a general-purpose Structure-from-Motion and Multi-View Stereo pipeline for ordered and unordered image collections. Verified 2026-07-11 from COLMAP documentation. This means the capture must support camera pose recovery and multi-view correspondence.

Production heuristics:

• Use fixed exposure, fixed white balance, fixed focus when practical.
• Avoid autofocus hunting, zoom changes, rolling-shutter whip pans, motion blur, and image stabilization jumps.
• Keep ISO low enough to preserve fine texture; add light or a tripod instead of accepting noisy, blurred images.
• Shoot RAW or high-quality stills where the pipeline supports them; keep originals unwarped.
• Maintain consistent scale and focal length groups. If a zoom lens changes focal length, split calibration groups by EXIF or capture segment.
• Use diffuse, stable lighting. Avoid moving shadows, specular glare, flash falloff, and screen flicker.
• Capture the subject in loops rather than from one panorama position. A panorama is usually weak for 3D reconstruction because the camera center barely translates.
• Keep each visible surface in at least two high-quality images; more views are better for hard geometry and occlusion.
• Capture coarse-to-fine: establish full-object or full-room loops, then add detail passes while preserving overlap to the broader set.
• Record a capture log while still on site: camera/lens, settings, route, scale references, permissions, weather, lighting, and known blind zones.

Official documentation facts from Agisoft and RealityScan capture guidance, verified 2026-07-11: high resolution is recommended; sharp images matter; low ISO reduces noise; f/8 to f/11 is a common depth-of-field recommendation in Agisoft guidance; slow shutter can create blur; aerial guidance commonly asks for about 80 percent forward and 60 percent side overlap; RealityScan guidance warns not to change viewpoint by more than about 30 degrees between photos and recommends moving around objects in complete loops. Treat these as official capture guidance, not universal physical law.

Overlap, exposure, focus, and motion

Overlap is not just a percentage. The usable overlap is the part of the subject that is sharp, exposed, textured, and shared with adjacent views.

Production heuristics:

• For objects: shoot at least two or three rings around the subject, plus top and low-angle passes where accessible.
• For interiors: make loops around room boundaries and islands, then add transitions through doorways and occluded corners.
• For exteriors and facades: capture broad establishing passes, oblique passes, and close detail strips.
• For aerial mapping-like capture: start from mission-planning overlap, then increase overlap for vegetation, low texture, oblique facades, altitude variation, or weak GNSS.
• Bracket only if the pipeline can handle or merge brackets predictably. Otherwise choose stable exposure that preserves texture in the subject.
• If motion cannot be controlled, shoot multiple passes and mask or remove moving objects from training/reconstruction.

Reject or isolate photos with heavy blur, clipped highlights on important texture, focus breathing mismatches, rain drops, lens smears, rolling-shutter distortion, people crossing the subject, or heavy JPEG artifacts.

Object capture

Use object capture for props, artifacts, furniture, food, products, costumes, sculptures, and specimens.

Production heuristics:

• Prefer a textured, matte, non-moving subject on a non-repetitive background unless masking is planned.
• If the object is reflective or textureless, use removable matte spray, cross-polarized light, projected texture, or a NeRF/3DGS-first deliverable when allowed by conservation and rights constraints.
• Capture scale with a ruler, scale bars, coded targets, or a measured turntable scene. Do not put the only scale marker on a surface that may be cropped away.
• If using a turntable, keep camera, lights, and background consistent and understand whether the solver expects the object or camera to move. Mask static backgrounds when the object rotates against them.
• Include underside and contact surfaces only if needed and if you can register them with sufficient overlap.

Environment and interior capture

Use environment capture for rooms, streets, cultural sites, sets, landscapes, and construction contexts.

Production heuristics:

• Build connected loops. Every side room, hallway, stair, or courtyard needs overlap back to the main path.
• Capture both human-eye height and lower or higher passes for occluded surfaces.
• For NeRF/3DGS, avoid large exposure jumps between windows and interiors; consider separate passes or controlled lighting if the viewer will inspect both.
• Record coordinate intent early: local artistic coordinates, building coordinates, GIS/geospatial, or surveyed control.
• Photograph control points, markers, and scale references clearly from multiple angles.

Aerial capture

Use aerial capture for roofs, terrain, quarries, facades, industrial sites, agriculture, and large cultural or architectural sites.

Production heuristics:

• Combine nadir and oblique imagery when vertical faces, facades, or overhangs matter.
• Maintain consistent altitude relative to the surface when possible; terrain variation may require adaptive flight planning.
• Place GCPs/checkpoints across the reconstruction area, including edges and elevation changes, when georeferencing or accuracy assessment matters.
• Respect aviation rules, site permissions, privacy, sensitive infrastructure restrictions, and wildlife or cultural-site restrictions before flying.
• Do not present drone EXIF GNSS alone as survey-grade accuracy.

People and performance capture

Use people capture only with explicit consent, privacy planning, and a clear use. People are difficult because faces, hair, clothing folds, and tiny motion all matter.

Production heuristics:

• Prefer multi-camera synchronized rigs for full-body/head capture; single-camera walkarounds risk expression drift and body sway.
• Use soft, even lighting; avoid blinking highlights and moving shadows.
• Capture neutral pose and expression unless a specific expression is requested.
• Treat biometric data, facial scans, tattoos, distinctive clothing, and minors as high-risk rights and privacy data.

Calibration, Scale, and Coordinates

A capture without scale is often visually useful but physically ambiguous.

Official documentation fact: Agisoft guidance states that reliable EXIF helps camera autocalibration, that missing EXIF can force assumptions, that fisheye/spherical/cylindrical cameras require appropriate camera type settings, and that cropping or geometrically transforming original images can affect autocalibration. Verified 2026-07-11.

Production workflow:

1. Preserve originals and EXIF. Never overwrite source images.
2. Group cameras by body, lens, focal length, focus setting, sensor mode, and preprocessing state.
3. Add measured scale bars, marker distances, surveyed targets, or GCPs before solving if scale matters.
4. Use checkpoints separate from control points when assessing geometric accuracy.
5. Decide coordinate frame before handoff: local origin, subject-centered origin, building coordinates, GIS CRS, or DCC/engine axis convention.
6. Document unit, up axis, forward axis, origin, scale source, georeferencing source, and any transform applied during export.

Do not say "accurate to X mm" unless the workflow includes independent check data and a measured error report. Say "scaled from two 500 mm bars; no independent accuracy guarantee" when that is the real state.

Reconstruction or Training Workflow

Mesh workflow

Typical mesh path:

1. Ingest originals and metadata.
2. Cull unusable images and split camera groups.
3. Detect features and match images.
4. Estimate camera poses and sparse points with SfM.
5. Add or refine calibration, scale, markers, and control.
6. Run dense reconstruction or depth-map estimation.
7. Build mesh or point cloud.
8. Clean, crop, fill only justified holes, decimate, unwrap, and texture.
9. Bake normals, ambient occlusion, displacement, or texture variants only when required by handoff.
10. Validate geometry, texture, scale, coordinates, and package.

Official documentation fact: COLMAP cites Structure-from-Motion Revisited (CVPR 2016) and Pixelwise View Selection for Unstructured Multi-View Stereo (ECCV 2016) for its SfM and MVS pipeline. Verified 2026-07-11 from COLMAP documentation.

NeRF workflow

Typical NeRF path:

1. Ingest source images or frames.
2. Estimate or import camera poses.
3. Remove blurred, inconsistent, duplicate, and dynamic-contaminated frames.
4. Define bounds, masks, near/far ranges, and scale normalization.
5. Train with held-out views for QA.
6. Render validation paths and compare with source views.
7. Export the runtime package, rendered video, depth, point cloud, or mesh extraction only if the target supports it.

Paper fact: NeRF needs known camera poses for optimization in the original formulation. If a user only has unordered images, pose estimation is part of the pipeline, not optional magic.

3D Gaussian splat workflow

Typical 3DGS path:

1. Estimate camera poses and

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: calesthio
• Source: calesthio/generative-media-skills
• License: MIT
• Homepage: https://github.com/calesthio/OpenMontage

Install and usage instructions live in the source repository linked above.