# Asset Harvester

> >-

- **Type:** Skill
- **Install:** `agentstack add skill-nvidia-nurec-skills-asset-harvester`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [NVIDIA](https://agentstack.voostack.com/s/nvidia)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [NVIDIA](https://github.com/NVIDIA)
- **Source:** https://github.com/NVIDIA/nurec-skills/tree/main/skills/asset-harvester

## Install

```sh
agentstack add skill-nvidia-nurec-skills-asset-harvester
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Asset Harvester

## Purpose

Install and drive NVIDIA Asset Harvester to extract per-object 3D
Gaussian Splat assets from sparse autonomous-vehicle (AV) object
observations — either a multi-view crop pulled from an NCore V4
driving log or a single masked image. The output is a
simulation-ready `gaussians.ply` plus optional `metadata.yaml` that
NVIDIA Omniverse NuRec can ingest as an external asset. Apache-2.0
upstream code lives at .

## When to Use / When NOT to Use

**Use this skill when:**

- The user has AV clips or masked single images and wants per-object
  3D assets via the SparseViewDiT + TokenGS pipeline.
- The user has NCore V4 driving-log clips and wants per-track 3D
  assets for simulation.
- The user asks about `SparseViewDiT`, `TokenGS`, or wants to
  reproduce the Asset Harvester paper / HF Space demo locally.
- The user wants `.ply` Gaussians + `metadata.yaml` suitable for
  NVIDIA Omniverse **NuRec** object insertion.

**Do NOT use this skill when:**

- The user wants a full-scene reconstruction (use the `nre` skill).
- The user has no per-object masks or AV-style object crops.
- The user wants text-to-3D, indoor scans, or non-AV imagery —
  out of distribution.
- The user wants to ingest raw sensor data into NCore V4 (use the
  `ncore` skill first).
- The user wants to re-train SparseViewDiT or TokenGS — this skill
  is install + inference only.
- The user just wants the no-install demo: point them at
  .

## Background

Open-source (Apache-2.0) image-to-3D pipeline pairing
**SparseViewDiT** (multiview diffusion, 16 consistent views) with
**TokenGS** (feed-forward Gaussian lifting):

```text
NCore V4 clip ──► NCore parsing ──► SparseViewDiT (16-view diffusion)
              ──► TokenGS lifting ──► gaussians.ply
              ──► (optional) metadata.yaml for NuRec object insertion
```

Single HF repo `nvidia/asset-harvester` ships four checkpoints:
`AH_object_seg_jit.pt` (AV-object Mask2Former),
`AH_multiview_diffusion.safetensors` (SparseViewDiT),
`AH_camera_estimator.safetensors` (camera pose, used when calibration
is absent), and `AH_tokengs_lifting.safetensors` (TokenGS).

## Inputs

- **image_root** — directory of per-object folders, each with
  `frame.jpeg` (512×512) and (optional) `mask.png` (required unless
  `component_store` is given).
- **component_store** — path to NCore V4 clip `.json` manifest,
  comma-separated component-store paths, or `.zarr.itar` glob
  (required when running the NCore parsing path).
- **output_dir** — where per-sample outputs (`gaussians.ply`,
  `multiview/`, `3d_lifted/`, `*.mp4`) are written (default
  `outputs/`).
- **offload_flag** — enable CPU offload (`--offload_model_to_cpu` /
  `--offload`) when VRAM ).

## Instructions

1. **Validate the host.** Have the agent execute
   `scripts/validate_setup.py` via its standard script runner —
   e.g. `run_script("scripts/validate_setup.py")` or
   `python scripts/validate_setup.py`. It confirms conda, the NVIDIA
   driver, GCC, and `HF_TOKEN` are in place and exits non-zero on
   any missing prerequisite. Do **not** print `$HF_TOKEN` directly;
   see [`references/installation.md`](references/installation.md).
2. **Install.** Use the one-shot `bash setup.sh` path unless the
   user asks for a manual install. Full commands and the pinned
   `gsplat` step are in
   [`references/installation.md`](references/installation.md).
3. **Download checkpoints.** `hf auth login` first, then
   `hf download nvidia/asset-harvester --local-dir checkpoints` (see
   [`references/installation.md`](references/installation.md)).
4. **Pick the inference path:**
   - Bundled demo → Workflow Q in
     [`references/workflows.md`](references/workflows.md).
   - Single user image (+/- mask) → Workflow S in the same file.
   - NCore V4 driving log → Workflow N (full walkthrough in
     [`references/end-to-end-ncore.md`](references/end-to-end-ncore.md)).
5. **Execute with appropriate VRAM flag.** If `/`.
7. **(Optional) Benchmark.** Clone the env to `av-object-benchmark`
   and run `benchmark/eval.py` for PSNR / LPIPS / SSIM and DINOv3
   embedding metrics. See
   [`references/end-to-end-ncore.md`](references/end-to-end-ncore.md).
8. **(Optional) Hand off to NuRec.** Rotate Gaussians with
   `orient_gaussians_for_nurec`, emit `metadata.yaml`, then follow
   the [NuRec external-assets docs](https://docs.nvidia.com/nurec/nurec/use-ah-assets.html).

## Examples

Three concrete entry points. Each one points at the workflow file
with the full command; nothing here is meant to be copy-pasted in
isolation.

### Example 1 — Smoke-test the install with bundled samples

```bash
python scripts/validate_setup.py          # then `bash setup.sh` once
python3 run_inference.py \
    --diffusion_checkpoint checkpoints/AH_multiview_diffusion.safetensors \
    --lifting_checkpoint   checkpoints/AH_tokengs_lifting.safetensors \
    --data_root            data_samples/rectified_AV_objects/ \
    --output_dir           outputs/harvesting
```

See Workflow Q in
[`references/workflows.md`](references/workflows.md).

### Example 2 — One masked single image → 3D asset

```bash
python -m asset_harvester.utils.image_segment \
    --checkpoint checkpoints/AH_object_seg_jit.pt \
    --image_folder data_samples/OOD_images
python3 run_inference.py \
    --diffusion_checkpoint checkpoints/AH_multiview_diffusion.safetensors \
    --ahc_checkpoint       checkpoints/AH_camera_estimator.safetensors \
    --lifting_checkpoint   checkpoints/AH_tokengs_lifting.safetensors \
    --image_dir            data_samples/OOD_images \
    --output_dir           outputs/single
```

See Workflow S in
[`references/workflows.md`](references/workflows.md).

### Example 3 — NCore V4 clip → NuRec-ready external assets

```bash
bash scripts/run_ncore_parser.sh --component-store 
bash run.sh --data-root ./outputs/ncore_parser --output-dir ./outputs/ncore_harvest
python -m asset_harvester.utils.orient_gaussians_for_nurec \
    --input-dir ./outputs/ncore_harvest \
    --output-dir ./outputs/ncore_harvest_nurec
python asset_harvester/utils/generate_external_assets_metadata.py \
    --input-dir ./outputs/ncore_harvest_nurec
```

Full walkthrough including sample-clip download, the benchmark
flow, and the NuRec PPISP caveat lives in
[`references/end-to-end-ncore.md`](references/end-to-end-ncore.md).

## Scripts

| Script | Purpose | Usage |
|--------|---------|-------|
| `scripts/validate_setup.py` | Verify host meets Asset Harvester prerequisites (conda, driver, GCC, `HF_TOKEN`). No network access. | Invoke via the agent's `run_script` helper, or `python scripts/validate_setup.py`. |

## Output Format

Per input sample (image or NCore track) the pipeline writes:

```text
${OUTPUT_DIR}//
├── multiview/           # 16 RGB views generated by SparseViewDiT
├── 3d_lifted/           # TokenGS-rendered views of the lifted Gaussians
├── gaussians.ply        # 3D Gaussian Splat asset (Omniverse/NuRec-ready)
├── multiview.mp4
└── 3d_lifted.mp4
```

When the NuRec handoff runs, `metadata.yaml` is additionally written
at the root of the oriented output directory.

## Prerequisites

Linux (Ubuntu 22.04 tested), conda, NVIDIA driver `>= 570` (CUDA
12.8), GCC 10–13, ~16 GB GPU VRAM, ~30 GB free disk, `HF_TOKEN` with
the `nvidia/asset-harvester` model card accepted, and egress to
`github.com`, `huggingface.co`, `pypi.org`,
`download.pytorch.org`. The check that fails-fast on a missing
prerequisite is `scripts/validate_setup.py`; secret-handling
guidance lives in
[`references/installation.md`](references/installation.md).

## References

- [`references/installation.md`](references/installation.md) —
  one-shot + manual install, checkpoint download, safe `HF_TOKEN`
  handling.
- [`references/workflows.md`](references/workflows.md) — Workflows
  Q (bundled), S (single image), N (NCore) plus a configuration
  matrix.
- [`references/end-to-end-ncore.md`](references/end-to-end-ncore.md)
  — full NCore V4 walkthrough including benchmark eval in the cloned
  `av-object-benchmark` env, and the NuRec handoff checklist.
- [`references/cli-reference.md`](references/cli-reference.md) —
  exhaustive flag matrix for `run_inference.py`, `run.sh`,
  `run_ncore_parser.sh`, `image_segment`,
  `orient_gaussians_for_nurec`,
  `generate_external_assets_metadata.py`.
- [`references/troubleshooting.md`](references/troubleshooting.md)
  — extended error matrix and full teardown / disk-cleanup recipe.
- Sibling skills: [`../ncore/SKILL.md`](../ncore/SKILL.md) (NCore V4
  ingest), [`../nre/SKILL.md`](../nre/SKILL.md) (NRE scene
  reconstruction and `export-external-assets` packaging),
  [`../physical-ai-datasets/SKILL.md`](../physical-ai-datasets/SKILL.md)
  (sample NCore clips, benchmark dataset).
- Upstream README: 
- Project page: 
- HF model: 
- Live demo: 
- NuRec external-assets guide: 

## Limitations

- AV-only domain. Non-road / non-AV objects are out of distribution.
- `AH_object_seg_jit.pt` is class-restricted (vehicles, VRUs,
  cyclists, road objects). Supply your own `mask.png` for arbitrary
  objects.
- Scale is **not predicted**. NuRec insertion reads scale from the
  source clip's cuboid tracks.
- 16 GB VRAM is the practical floor; lower-VRAM users must offload
  to CPU (slower).
- Inputs must be 512×512 square crops.
- `benchmark/eval.py` needs a separately cloned conda env
  (`av-object-benchmark`) because `transformers>=4.56.0` conflicts
  with the main env's pinned `transformers==4.48.3`.
- Linux-only install path (tested on Ubuntu 22.04 + CUDA 12.8).
- Optional SAM 3D Body metric needs the gated
  `facebook/sam-3d-body-dinov3` repo; eval falls back to
  PSNR/LPIPS/SSIM if unavailable.

## Troubleshooting (top 4)

| Error | Cause | Solution |
|-------|-------|----------|
| `gsplat` import / CUDA ABI mismatch | Installed `gsplat` from PyPI wheel instead of the pinned commit | Reinstall from the pinned source commit; see [`references/installation.md`](references/installation.md). |
| `nvcc` "unsupported GNU version" | GCC outside 10–13 on PATH | Install GCC 12 and export `CC`/`CXX`/`CUDAHOSTCXX` before `setup.sh`. |
| `CUDA error: out of memory` | GPU VRAM < ~16 GB | Add `--offload_model_to_cpu` (direct) or `--offload` (`run.sh`). |
| `401 Unauthorized` from `hf download` | Model-card terms not accepted, or missing `HF_TOKEN` | Accept the model card and re-run `hf auth login`. |

Full matrix + teardown live in
[`references/troubleshooting.md`](references/troubleshooting.md).

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [NVIDIA](https://github.com/NVIDIA)
- **Source:** [NVIDIA/nurec-skills](https://github.com/NVIDIA/nurec-skills)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-nvidia-nurec-skills-asset-harvester
- Seller: https://agentstack.voostack.com/s/nvidia
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.