Physical Ai Datasets

NVIDIA Physical AI Datasets (Hugging Face)

Purpose

Help the agent find, evaluate, and download an NVIDIA PhysicalAI-* dataset on Hugging Face that fits the user's downstream task — autonomous-vehicle reconstruction, robotics manipulation, GR00T post-training, spatial-intelligence research, grasping, or sim-ready content — and then hand off to the sibling skill (ncore, nre, asset-harvester, nurec-fixer) or upstream NVIDIA tool that actually consumes it.

Use this skill when: the user asks "is there an NVIDIA dataset for X?", "where do I get NCore / NuRec / GR00T sample data?", or is shopping the Hugging Face NVIDIA org for PhysicalAI-* collections.

Do NOT use this skill when:

• The user already knows the dataset and just wants to run a

pipeline — jump straight to the consuming skill.

• The user needs a non-NVIDIA dataset (Waymo, nuScenes, KITTI, …) —

this catalog is NVIDIA-only.

• The user wants to train Cosmos / GR00T / Isaac Sim itself — that's

the upstream tool's job, not this catalog's.

Overview

Catalog of NVIDIA's open Physical AI dataset family on Hugging Face. Pick by task (Section 2 § lookup table) or family (Sections 3–10). Every entry lists: dataset path, size, format, license, gating, and the downstream skill in this repo that consumes it.

> Source of truth: (filter PhysicalAI-) > and the curated Physical AI collection. > When upstream cards drift, re-check the HF page; this skill mirrors > the cards as of Apr 2026.

Prerequisites

• HuggingFace account with the **dataset card opened in a browser at

least once**, and the gating checkbox accepted on every dataset you intend to download.

• HuggingFace user access token exported as HF_TOKEN (create at

).

• git, git-lfs, and huggingface_hub[cli] on PATH.
• Storage room sized to the dataset you're pulling (see the per-row

size column; some are whenever HF_TOKEN is set, because ${VAR:-no} only falls back to "no" when VAR is empty — when set it expands to $VAR`. Use one of these instead:

hf auth whoami                              # confirms the token without echoing it
test -n "$HF_TOKEN" && echo "HF_TOKEN: set (${#HF_TOKEN} chars)" || echo "HF_TOKEN: missing"

Rotate any token you suspect was echoed at .

Table of Contents

1. [Common download recipe](#common-download-recipe) — HF auth, gating, CLI.
2. [Filtered AV download recipe](#filtered-av-download-recipe) — default hyperion_8.1 filter for PhysicalAI-Autonomous-Vehicles raw pulls.
3. [Pick a dataset by task](#pick-a-dataset-by-task) — fast lookup table.
4. [Autonomous Vehicles](#autonomous-vehicles) — 5 datasets.
5. [Robotics — Manipulation](#robotics--manipulation) — 6 datasets.
6. [Robotics — GR00T](#robotics--gr00t) — 7 datasets.
7. [Robotics — mindmap](#robotics--mindmap) — 4 datasets.
8. [Robotics — NuRec / Sim-Ready scenes](#robotics--nurec--sim-ready-scenes) — 2 datasets.
9. [Robotics — Healthcare](#robotics--healthcare) — 1 dataset.
10. [Robotics — Grasping](#robotics--grasping) — 1 dataset.
11. [Robotics — Physical / material properties](#robotics--physical--material-properties) — 2 datasets.
12. [Spatial Intelligence + SimReady scenes](#spatial-intelligence--simready-scenes) — 5 datasets.
13. [Community / sample](#community--sample) — 1 dataset.
14. [License decision tree](#license-decision-tree) — what you can do with each.
15. [Cross-skill usage map](#cross-skill-usage-map) — which skill consumes which dataset.

Common download recipe

All NVIDIA Physical AI datasets live on huggingface.co/datasets/nvidia/... and use the same access shape:

sudo apt -y install git git-lfs
git lfs install

uv tool install -U "huggingface_hub[cli]"   # or: pip install --upgrade "huggingface_hub[cli]"
hf auth login                                # paste user access token

The token must:

1. Have a HF user account that's logged in to the dataset page in a browser AT LEAST ONCE.
2. Have accepted any license / terms-of-use checkbox the dataset shows

(re-accept if the dataset has been re-gated — common for AV).

Three download patterns:

# Whole dataset (small / medium)
hf download nvidia/ --repo-type dataset --local-dir ./

# Sub-folder only (recommended for large multi-task collections)
hf download nvidia/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim \
  --repo-type dataset \
  --include "gr1_arms_only.CanSort/**" \
  --local-dir ./gr00t_dataset

# Sparse-checkout via git-LFS (if you want incremental git-style work)
git clone --filter=blob:none --no-checkout https://huggingface.co/datasets/nvidia/
cd 
git sparse-checkout init --cone
git sparse-checkout set 
git checkout main

Special-case downloaders:

• PhysicalAI-Autonomous-Vehicles (133 TB) — use the official Python

toolkit pip install physical_ai_av (NVlabs/physicalai_av) to filter by sensor / country / split before downloading; otherwise you will pull TBs you don't need.

Default policy: pre-filter to platform_class == 'hyperion_8.1' before any bulk pull. The dataset mixes two sensor rigs (hyperion_8 and hyperion_8.1) and the entire downstream NuRec / NCore / Asset-Harvester chain in this repo is only validated for hyperion_8.1. Roughly half of the 306k clips fall outside that platform, so the filter typically cuts the working set (and download size) in half.

Apply this filter by default in any download script you generate. Only skip the filter when the user explicitly asks for it (e.g. "download all platforms", "include hyperion_8", "don't filter by platform") or when they hand you a specific clip UUID — in that single-clip case, download_clip_features(clip_id=...) is already scoped and no platform filter is needed.

Canonical filtered recipe (see § [Filtered AV download recipe](#filtered-av-download-recipe) below for an end-to-end example):

```python from physicalaiav import PhysicalAIAVDatasetInterface

dataset = PhysicalAIAVDatasetInterface() dataset.downloadmetadata() dc = dataset.metadata['datacollection'] hyperion81clipids = dc[dc['platformclass'] == 'hyperion_8.1'].index.tolist() ```

• PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams (3 TB) — use the

upstream download.py (nv-tlabs/Cosmos-Drive-Dreams) with --file_types {hdmap,lidar,synthetic} to select layers.

• PhysicalAI-SpatialIntelligence-Lyra-SDG (25 TB) — `hf download

--local-dir lyra_dataset/tar`; untar each tar yourself.

• Spatial-Intelligence-Warehouse — chunked TAR-GZs need a manual

loop after download (script provided in the upstream card).

For dataset filtering / preview: NVIDIA's Cosmos Dataset Search (CDS) lets you query a 41K subset of the AV dataset semantically before downloading.

Filtered AV download recipe

Always use this recipe (or a derivative of it) when the user asks to download raw clips from PhysicalAI-Autonomous-Vehicles. The hyperion_8.1 filter is the default; it matches every downstream skill in this repo (ncore, nre, asset-harvester, nurec-fixer).

When to skip the platform filter:

1. The user explicitly opts out — e.g. "download all platforms",

"include hyperion8 too", "ignore platformclass", or asks for a dataset-wide statistic. In that case, drop the platform mask and warn them that NuRec / NCore tooling will not work on the hyperion_8 clips.

1. The user gave you a specific clip UUID. A single-clip

download_clip_features(clip_id=...) is already scoped, and filtering by platform_class for one clip is pointless.

Recipe (defaults: filter on; user may layer extra masks like country or sensor presence on top):

from physical_ai_av import PhysicalAIAVDatasetInterface

dataset = PhysicalAIAVDatasetInterface()

dataset.download_metadata()
dc = dataset.metadata['data_collection']

clip_mask = dc['platform_class'] == 'hyperion_8.1'

# Optional extra masks (only add when the user asked for them):
# clip_mask &= dc['country'] == 'US'
# sp = dataset.metadata['feature_presence']   # 26.03+; was 'sensor_presence' in 25.10
# clip_mask &= sp['lidar_top_360fov']

clip_ids = dc[clip_mask].index.tolist()
print(f"Downloading {len(clip_ids)} hyperion_8.1 clips")

dataset.download_clip_features(
    clip_id=clip_ids,
    features=["camera_front_wide_120fov", "lidar_top_360fov", "egomotion"],
    max_workers=8,
)

Single-clip fast path (no platform filter — the UUID is already specific):

dataset.download_clip_features(
    clip_id="",
    features=["camera_front_wide_120fov", "lidar_top_360fov", "egomotion"],
)

Explicit opt-out (only when the user asked for it):

clip_ids = dc.index.tolist()   # NO platform filter — all 306k clips
# WARNING: downstream NuRec / NCore / Asset-Harvester only handle
# the hyperion_8.1 subset.

Pick a dataset by task

| Goal | Recommended dataset(s) | |------|------------------------| | End-to-end AV training (real, multi-sensor) | PhysicalAI-Autonomous-Vehicles (133 TB, 1700 h, 25 countries) | | AV in NCore V4 format (drop-in for [ncore](../ncore/SKILL.md)) | PhysicalAI-Autonomous-Vehicles-NCore (~1.1k clips) | | AV photoreal Sim2Real / weather augmentation | PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams (3 TB; 7 weather variants) | | AV neural reconstructions ready for CARLA / NuRec | PhysicalAI-Autonomous-Vehicles-NuRec (918 USDZ scenes) | | Asset Harvester / 3DGS extraction sample clip | PhysicalAI-Autonomous-Vehicles-NCore | | GR00T post-training, broad coverage | PhysicalAI-Robotics-GR00T-X-Embodiment-Sim (1.91 TB, 24 GR1 task families + bimanual + RoboCasa) | | GR00T fine-tune on industrial tasks | PhysicalAI-GR00T-Tuned-Tasks (Nut Pouring, Exhaust Pipe Sorting) | | GR00T eval images / videos | PhysicalAI-Robotics-GR00T-Eval, PhysicalAI-Robotics-GR00T-GR1 | | Real humanoid teleop (Unitree G1) | PhysicalAI-Robotics-GR00T-Teleop-G1 (1000 trajectories) | | Sim humanoid teleop (Fourier GR1) | PhysicalAI-Robotics-GR00T-Teleop-Sim (24 tasks × 1k trajectories) | | Massive humanoid pretraining (44k h, DreamDojo) | PhysicalAI-Robotics-GR00T-Teleop-GR1 (74.3 GB) | | Spatial-memory imitation learning (mindmap) | PhysicalAI-Robotics-mindmap-{Stick-in-Bin,Drill-in-Box,Cube-Stacking,Mug-in-Drawer} | | Robot pick-place in kitchen (bimanual Kinova Gen3) | PhysicalAI-Robotics-Manipulation-Kitchen, -Manipulation-Objects | | Robot pick-place tabletop (single Franka) | PhysicalAI-Robotics-Manipulation-SingleArm | | Cosmos-Transfer1 visual-augmented stacking | PhysicalAI-Robotics-Manipulation-Augmented | | Massive teleop in kitchen (Franka + mobile base) | PhysicalAI-Robotics-Manipulation-Kitchen-Demos (600 h, 316 tasks, 55k traj) | | MJCF kitchen objects + fixtures (MuJoCo) | PhysicalAI-Robotics-Manipulation-Objects-Kitchen-MJCF | | Sim-Ready warehouse for IsaacSim | PhysicalAI-SimReady-Warehouse-01 (753 USD assets) | | GR1 tabletop digital cousins (assets) | PhysicalAI-DigitalCousin-Assets | | 3DGS / Sim-Ready indoor scenes for AMR sim | PhysicalAI-Robotics-NuRec (Nova Carter labs, Zurich offices, hand-held) | | Multi-cam tracking + 3D box benchmark | PhysicalAI-SmartSpaces (AI City Challenge 2024 + 2025) | | 3D scene QA / VLM training (warehouses) | PhysicalAI-Spatial-Intelligence-Warehouse (499k QA pairs) | | Generative 3D scene reconstruction training | PhysicalAI-SpatialIntelligence-Lyra-SDG (25 TB; GEN3C-derived) | | Radiance-field photometric benchmark | PhysicalAI-NuRec-PPISP (8 sequences, +/-2 EV bracketing) | | Grasping models (Franka, Robotiq-2f-140, suction) | PhysicalAI-Robotics-GraspGen (57M grasps, Objaverse-LVIS) | | Healthcare / surgical robotics autonomy | PhysicalAI-Robotics-Open-H-Embodiment (750 h, 4.5 TB, 30+ orgs) | | Volumetric mechanical / material properties | PhysicalAI-Robotics-PhysicalAssets-VoMP, -VoMP-Eval |

Autonomous Vehicles

PhysicalAI-Autonomous-Vehicles

The flagship real-world AV dataset.

| Field | Value | |-------|-------| | HF | | | Size | 133 TB — 306,152 clips × 20 s = 1700 h | | Sensors | 7 cameras (1080p @ 30 FPS), top-360 LiDAR (10 Hz, 298k clips), up to 10 radars (160k clips), ego motion, calibration, machine labels | | Geography | 25 countries, 2500+ cities (US 155k, Germany 44k, …) | | Format | Per-sensor parquet/mp4 chunks of ~100 clips; UUIDs cross-link sensors | | License | NVIDIA AV Dataset License Agreement (gated; AV-development-only purpose; no biometric / surveillance / re-identification; expires 12 months after download) | | Toolkit | pip install physical_ai_av — direct filtered downloads + format docs. Default to filtering metadata['data_collection']['platform_class'] == 'hyperion_8.1' before any bulk pull (see [Filtered AV download recipe](#filtered-av-download-recipe)); only skip the filter on explicit user request or when downloading a specific clip UUID. | | Versions | 26.03 (current; offline-optimized features for 97 % of clips), 25.10 (initial) | | Subset preview | 41k clips searchable on Cosmos Dataset Search | | Use with | [../ncore/SKILL.md](../ncore/SKILL.md) (convert raw clips to NCore V4), [../asset-harvester/SKILL.md](../asset-harvester/SKILL.md) (extract per-object Gaussian assets). NuRec workflows are only validated for platform_class == hyperion_8.1. Upstream sim/training tools without an in-repo skill: NVlabs/alpamayo-1.5, NVlabs/alpasim, CARLA. |

PhysicalAI-Autonomous-Vehicles-NCore

Curated NCore V4 subset of the above.

| Field | Value | |-------|-------| | HF | | | Size | ~1.1k clips with accurate offline calibration / egomotion / cuboids | | Format | NCore V4 — pai_.json + per-sensor .zarr.itar files | | License | NVIDIA AV Dataset License Agreement (gated, same as above) | | Use with | [../ncore/SKILL.md](../ncore/SKILL.md) (drop-in), [../asset-harvester/SKILL.md](../asset-harvester/SKILL.md) (sample clip path: clips/2a6f330-5ab0-4e92-99d4-d19e406952f4/) | | Notes | Built via PAI data converter. Use this BEFORE the full AV dataset for any NCore-driven workflow. |

PhysicalAI-Autonomous-Vehicles-NuRec

Pre-built NuRec dynamic neural reconstructions ready for IsaacSim / CARLA.

| Field | Value | |-------|-------| | HF | | | Size | 918 USDZ scenes, ~20 s each, with surface meshes + front-camera mp4 + labels.json (Batch0002+) | | Reconstruction | 6 cameras (front-wide 120°, front-tele 30°, cross-L/R 120°, rear-L/R 70°) | | Versions | 26.02 (current), 25.07, 25.05 | | License | NVIDIA AV Dataset License Agreement (gated) | | Use with | [../nre/SKILL.md](../nre/SKILL.md) (render the USDZs locally or over serve-grpc), [../nurec-fixer/SKILL.md](../nurec-fixer/SKILL.md) (clean up rendered frames). Upstream consumer without an in-repo skill: CARLA (NuRec integration in 0.9.16+). |

PhysicalAI-Autonomous-Vehicle-Cosmos-Drive-Dreams

Cosmos-Transfer-style synthetic + HD-map labels for diverse weather.

| Field | Value | |-------|-------| | HF | | | Size | 3 TB total (synthetic only ~700 GB) — 5,843 RDS-HQ clips × 2 chunks × 7 weather = 81,802 synthetic videos (121 frames each) | | Modalities | Cosmos-generated MP4, HDMap (lanes/lanelines/road boundaries/wait lines/crosswalks/markings/poles/lights/signs), LiDAR, vehicle pose, camera intrinsics (ftheta + pinhole), 4D object tracking | | Cameras | 7 (front-wide/cross-L/cross-R/rear-L/rear-R/rear-tele/front-tele) | | Weather variants | Foggy / Golden hour / Morning / Night / Rainy / Snowy / Sunny | | License | CC-BY-4.0 (commercial OK with attribution) | | Tooling | `wget … scripts/downloa

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: NVIDIA
• Source: NVIDIA/nurec-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.