Ncore

NCore V4 Data Conversion

Purpose

Convert any sensor recording (cameras, LiDAR, radar, IMU, depth, stereo, COLMAP/SfM, ROS 2 bag) into a valid NVIDIA NCore V4 store so it can be consumed by NuRec / Asset Harvester / ncore_vis, or wired into a robotics-to-sim ("r2s") pipeline. Drive the existing in-tree converters (PAI, Waymo, COLMAP/ScanNet++) or author a new converter from ncore_template/.

Use this skill when: the user has raw sensor data (any rig) that NuRec or Asset Harvester needs to ingest, or when an existing converter is failing validate.py / producing NuRec data-quality complaints.

Do NOT use this skill when:

• The user is already on V4 and only wants to train or render

(use the nre skill).

• The user wants per-object 3D assets from sparse views (use

asset-harvester).

• The user only needs to browse / pick an existing NVIDIA dataset

(use physical-ai-datasets).

This skill teaches an agent to take any sensor dataset and produce a valid NCore V4 store that NuRec / Asset Harvester / ncore_vis will accept. It covers both driving the existing in-tree converters (PAI, Waymo, COLMAP/ScanNet++) and writing a new one for unsupported formats (PandaSet, NuScenes, KITTI, stereo, mono+depth, mono+lidar, custom robotics rigs).

Table of Contents

1. [When to use which path](#when-to-use-which-path)
2. [Install & references](#install--references)
3. [Mental model — the V4 store](#mental-model--the-v4-store)
4. [Path A — drive an existing in-tree converter](#path-a--drive-an-existing-in-tree-converter)
5. [Path B — author a new converter from the template](#path-b--author-a-new-converter-from-the-template)
6. [V4 conventions you must obey](#v4-conventions-you-must-obey)
7. [Format recipes (AV)](#format-recipes-av)

• [PAI (NVIDIA Physical AI Autonomous Vehicles, HuggingFace)](#pai-nvidia-physical-ai-autonomous-vehicles-huggingface)
• [Waymo Open](#waymo-open)
• [PandaSet](#pandaset)
• [NuScenes](#nuscenes)

1. [Format recipes (non-AV / sensor-only)](#format-recipes-non-av--sensor-only)

• [Mono camera (no depth, no LiDAR) → COLMAP track](#mono-camera-no-depth-no-lidar--colmap-track)
• [Stereo cameras](#stereo-cameras)
• [Multi-stereo rig (surround stereo)](#multi-stereo-rig-surround-stereo)
• [Mono + depth (RGB-D / learned depth)](#mono--depth-rgb-d--learned-depth)
• [Mono + LiDAR (handheld / robot)](#mono--lidar-handheld--robot)
• [Solid-state / non-repetitive LiDAR (Livox)](#solid-state--non-repetitive-lidar-livox)
• [IMU + camera (visual-inertial)](#imu--camera-visual-inertial)
• [ROS2 bag (MCAP / SQLite3)](#ros2-bag-mcap--sqlite3)
• [Aerial / drone](#aerial--drone)

1. [Robotics pipeline shards (r2s)](#robotics-pipeline-shards-r2s)
2. [Validation & end-to-end NuRec](#validation--end-to-end-nurec)
3. [Common failure modes (and the fix file)](#common-failure-modes-and-the-fix-file)
4. [Additional resources](#additional-resources)

---

When to use which path

| You have… | Use | |-----------|-----| | PAI clip on HuggingFace, or a local PAI clip directory | Path A — tools/data_converter/pai:convert (pai-stream-v4 / pai-v4) | | Waymo .tfrecord files | Path A — tools/data_converter/waymo:convert (waymo-v4) | | COLMAP scene (or ScanNet++ DSLR) | Path A — tools/data_converter/colmap:convert (colmap-v4 / scannetpp-v4) | | Mono RGB images, no poses | Path A.5 — run COLMAP first, then colmap-v4 | | PandaSet, NuScenes, KITTI, Argoverse, custom AV rig | Path B — author from ncore_template/impl/data_converter/example_converter.py | | Stereo / mono+depth / mono+lidar / robotics | Path B | | Already have parsed numpy/torch arrays in memory | Path B but skip Bazel — use ncore.data.v4 API directly |

If a candidate path exists in upstream, prefer it. Hand-rolling a Waymo or PAI converter on top of the template is wasted work and almost certainly wrong (rolling-shutter timing, FTheta intrinsics, Waymo camera-frame rotation, etc).

---

Prerequisites

• Linux host with Python ≥ 3.10 and pip.
• git for the upstream converter sources.
• bazel only if you run the in-tree converters (Path A); pure-Python

in-process writes (ncore.data.v4) need no Bazel.

• Disk: budget tens of GB per converted clip; pre-zarr scratch can be

larger than the final .zarr.itar.

• HuggingFace token (HF_TOKEN) only if pulling a gated PAI clip.

Verifying secrets safely

Always verify prerequisites with the upstream validate.py or by running the converter against a tiny test slice; never write ad-hoc bash that interpolates HF_TOKEN values. The common one-liner

# BAD — leaks the secret to the terminal when the variable is set
echo "HF_TOKEN: ${HF_TOKEN:+yes}${HF_TOKEN:-no}"

prints yes whenever HF_TOKEN is set, because ${VAR:-no} only falls back to "no" when the variable is empty. Use a length-only check, which never echoes the value:

# OK — prints "set (N chars)" or "missing", never the value
test -n "$HF_TOKEN" && echo "HF_TOKEN: set (${#HF_TOKEN} chars)" || echo "HF_TOKEN: missing"

Rotate any token you suspect was echoed at .

Install & references

pip install nvidia-ncore        # pure-Python API for in-process writes
git clone --depth 1 https://github.com/NVIDIA/ncore.git   # for upstream converters (Bazel)

• Source + upstream converters:
• V4 spec / conventions:
• API reference:
• Conversion guides:
• Sensor models (camera, LiDAR, windshield):
• The template scaffold (every method documented inline):

[ncore_template/impl/data_converter/example_converter.py](ncoretemplate/impl/dataconverter/example_converter.py)

• End-to-end NCore → NRE training and rendering: see the sibling

[nre](../nre/SKILL.md) skill (Workflow A). NVIDIA's reference OSMO recipe that wires PAI → NCore → NuRec → USDZ training lives in the upstream NCore repo at

and the NRE container docs at .

---

Mental model — the V4 store

Every V4 sequence is one store (itar archive or plain directory) holding component groups. The required components for NuRec are:

| Component | What it carries | API class | |-----------|-----------------|-----------| | Poses | Dynamic T_rig_world (per timestamp); static T_sensor_rig per camera/lidar/radar; static T_world_world_global | PosesComponent | | Intrinsics | Per-camera model (Pinhole / Fisheye / FTheta) + per-LiDAR spinning model | IntrinsicsComponent | | CameraSensor | Encoded image bytes + per-frame [exposure_start, exposure_end] µs | CameraSensorComponent | | LidarSensor | Per-ray unit direction + per-ray µs timestamp + distance(s) + intensity + model_element=(row,col) | LidarSensorComponent | | RadarSensor (optional) | Same shape as LiDAR minus intensity / model_element | RadarSensorComponent | | Cuboids (optional, recommended) | CuboidTrackObservation list referencing rig / world / sensor frames | CuboidsComponent | | Masks | Per-camera dict of {name: PIL.Image} (NuRec requires ego masks) | MasksComponent | | PointClouds (optional) | Pre-computed dense or SfM points (e.g. COLMAP sfm_points, depth-derived) | PointCloudsComponent |

Frames of reference (these are non-negotiable — wrong frames = silent NuRec failure):

• Rig: +X forward, +Y left, +Z up. Origin at the middle of the rear axle on nominal ground for AV, or any natural body-fixed point for non-AV. All extrinsics are T_sensor → rig.
• Camera sensor: +X right, +Y down, +Z forward (optical axis). NCore's convention. Waymo (X-fwd) and similar must be rotated before storing extrinsics.
• LiDAR model frame: azimuth 0° = +X, 90° = +Y, +Z up. Independent of the raw sensor's native azimuth — you choose how column_azimuths_rad maps physical columns.
• World: sequence-local. Re-reference all T_rig_world to the first ego pose so origin is near the vehicle start (raw UTM/ECEF at 10+ km loses precision in float32). Carry the original first pose into T_world_world_global (float64) if you need a global anchor.
• Image pixels: u right, v down, origin at the top-left corner of the top-left pixel (so pixel centers are at 0.5, 0.5).
• Units: timestamps in µs everywhere, distances in metres, angles in radians.

The single source of truth is the spec — when in doubt, open it: .

Sequence-level metadata

Some downstream pipelines need metadata beyond the per-component data — carry it on the sequence's generic_meta_data (passed to SequenceComponentGroupsWriter(...)):

• Stereo pairs — required by stereo-depth modules (Foundation Stereo) and

multi-camera training configs to discover left/right pairings:

``json {"stereo_pairs": [{"left": "camera_front_left", "right": "camera_front_right"}]} ``

Multiple pairs are allowed for surround-stereo rigs.

• Source tag — distinguishes real from synthetic data. Renderer-output

shards (NuRec sim) set {"source": "simulation", "model_checkpoint": "..."}; real-sensor shards omit the key or set {"source": "real"}. Downstream validators key off this to skip "expected vs measured" checks on sim data.

• Calibration / egomotion provenance — the example template writes

calibration_type and egomotion_type on the PosesComponent's generic_meta_data. Use this to track the upstream tool (e.g. egomotion_type: "cuvslam-stereo" or "kiss-icp" or "vio:orbslam3") so later modules can pick refinement strategies that match the input quality.

---

Instructions

Pick one of the two paths below.

• Path A if the user's dataset format is already supported in

ncore/tools/data_converter/ (PAI, Waymo, COLMAP/ScanNet++) — bootstrap the upstream repo and drive the existing binary.

• Path B if the format is unsupported (PandaSet, NuScenes, KITTI,

custom rig, robotics bag, …) — copy ncore_template/ next to the dataset and fill in the four hand-written hooks. The V4 conventions in the Mental model section above are mandatory; the recipes further down show typical configurations per rig.

After conversion, always run the Validation & end-to-end NuRec section below before handing the store to NRE.

Path A — drive an existing in-tree converter

The upstream tools/data_converter/ modules are Bazel targets. Build once, run per dataset.

Bootstrap

git clone --depth 1 https://github.com/NVIDIA/ncore.git
cd ncore
bazel build //tools/data_converter/pai:convert     # or waymo, or colmap

Each convert binary takes shared base flags (--root-dir, --output-dir, --no-cameras, --camera-id, --no-lidars, --lidar-id, --verbose) followed by a subcommand (pai-v4, pai-stream-v4, waymo-v4, colmap-v4, scannetpp-v4) with format-specific flags.

Standard sub-flags worth knowing

| Flag | Default | Meaning | |------|---------|---------| | --store-type {itar,directory} | itar | itar is fastest for NuRec; directory is debuggable | | --profile {default,separate-sensors,separate-all} | varies | NuRec wants separate-sensors | | --sequence-meta / --no-sequence-meta | enabled | Writes .json next to the store — NuRec/ncore_vis need it | | --world-global-mode {none,identity,localized} | varies | For NuRec releases that require the world→world_global edge, use identity (or localized to keep a real global anchor) |

When to script vs run interactively

For a one-off conversion, the bare bazel run form in each format recipe below is enough. For repeatable cluster runs, wrap the same two steps (bazel build then bazel run) inside an OSMO / Slurm / Kubernetes task that clones NCore at a pinned ref, runs the convert step, and chains the result into the [nre](../nre/SKILL.md) training and aux-data containers (see nre's Workflow A). The upstream NVIDIA/ncore repo ships reference converter targets that you can pin by Git commit for reproducibility.

---

Path B — author a new converter from the template

The scaffold is intentionally minimal but writes every required component type with placeholder data. Treat it as a checklist: every # FILL IN is a correctness gate — none can be skipped.

Scaffold

# Copy the scaffold next to your dataset (run from this skill's folder)
cp -r ncore_template /path/to/ncore-myformat

Then rename the package and class (ExampleConverter → MyFormatConverter) and implement the contract:

| Method | Contract | |--------|----------| | get_sequence_ids(config) -> list[str] | Discover sequence IDs from config.root_dir (or wherever your dataset lives — manifest CSV, HF clip index, ROS bag glob) | | from_config(config) -> Converter | One-time setup (load calibration, open dataset index, init shared interpolators). Heavy lifting that all sequences share goes here | | convert_sequence(sequence_id) -> None | Per-sequence work: open SequenceComponentGroupsWriter, register component writers, write data, finalize(), write .json |

Inside convert_sequence the canonical order is Poses → Intrinsics → Masks → Camera → LiDAR → Radar → Cuboids → finalize. This order is not required by the writer but it surfaces calibration / pose / timing bugs before you've spent minutes encoding image bytes.

The skeleton walks each step explicitly and lists every silent-correctness trap inline — read these before filling them in:

• Spinning-LiDAR pitfalls (spinning_direction, non-uniform row_elevations_rad,

column_azimuths_rad ordering, Ouster row_azimuth_offsets_rad): [example_converter.py:61-120](ncoretemplate/impl/dataconverter/example_converter.py#L61-L120)

• Pose trajectory density + float64 → float32 + re-referencing rules:

[example_converter.py:336-492](ncoretemplate/impl/dataconverter/example_converter.py#L336-L492)

• Camera intrinsics for Pinhole / Fisheye / FTheta + shutter type:

[example_converter.py:524-578](ncoretemplate/impl/dataconverter/example_converter.py#L524-L578)

• Per-ray LiDAR timestamps and the three data shapes (range image / sensor-frame

XYZ / world-frame XYZ requiring decompensation): [example_converter.py:746-829](ncoretemplate/impl/dataconverter/example_converter.py#L746-L829)

• Cuboid centroid convention (geometric center, not bottom-center):

[example_converter.py:876-980](ncoretemplate/impl/dataconverter/example_converter.py#L876-L980)

In-process API (no Bazel)

If you already have parsed arrays in Python and don't need a CLI, skip FileBasedDataConverter entirely and call the V4 writer directly:

from ncore.data.v4 import SequenceComponentGroupsWriter, PosesComponent, ...
writer = SequenceComponentGroupsWriter(
    output_dir_path=out / seq_id,
    store_base_name=seq_id,
    sequence_id=seq_id,
    sequence_timestamp_interval_us=interval,
    store_type="itar",
)
poses_writer = writer.register_component_writer(PosesComponent.Writer, ...)
# … write each component …
paths = writer.finalize()

The contract (component order, dtype rules, timestamp constraints) is identical.

---

V4 conventions you must obey

These are the rules that turn into runtime asserts (or worse: silent NuRec artefacts). Cross-reference the spec before relaxing any of them.

Time

• Microseconds, uint64, everywhere. np.uint64, not np.int64.
• The sequence interval is half-closed [start, stop). Build it with

HalfClosedInterval.from_start_end(start, end_inclusive) — do not pre-add 1 to end.

• Dynamic poses must exactly span the interval: timestamps[0] == start and

timestamps[-1] == stop - 1. Sensors with timestamps slightly outside this range are clamped at write time.

• Per-sensor frame timestamps are [exposure_start, exposure_end] (cameras) or

[sweep_start, sweep_end] (LiDAR/radar). They must lie within the sequence interval and the end m

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: NVIDIA
• Source: NVIDIA/nurec-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.