# Raw Hdf5 Standardization

> Standardize raw research data into metadata-rich HDF5 files and update Python simulation or measurement code to export consistent raw HDF5 datasets. Use when Codex needs to reorganize raw csv/txt/dat/npy data, add metadata, design HDF5 schemas, enforce raw-data naming conventions, validate HDF5 structure, preserve original raw files while creating standardized copies, or answer Chinese requests s…

- **Type:** Skill
- **Install:** `agentstack add skill-narroog-research-skills-raw-hdf5-standardization`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Narroog](https://agentstack.voostack.com/s/narroog)
- **Installs:** 0
- **Category:** [Data & Analytics](https://agentstack.voostack.com/c/data-and-analytics)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Narroog](https://github.com/Narroog)
- **Source:** https://github.com/Narroog/research-skills/tree/main/skills/raw-hdf5-standardization

## Install

```sh
agentstack add skill-narroog-research-skills-raw-hdf5-standardization
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Raw HDF5 Standardization Skill

## Purpose

This skill standardizes raw research data into metadata-rich HDF5 files and updates existing code so that future raw data is exported directly in the same format.

The goal is to make raw research data:

- traceable
- reproducible
- consistently named
- machine-readable
- suitable for long-term storage
- easy to convert into processed CSV files for plotting

This skill is mainly for raw data governance. It is not a plotting skill. Processed plotting data should usually be exported separately as CSV.

---

## When to Use This Skill

Use this skill when the user asks to:

1. reorganize existing raw data files into HDF5 format
2. add metadata to raw data
3. rename raw data files according to a project convention
4. modify Python simulation or measurement code to export HDF5
5. create reusable raw-data export utilities
6. clean inconsistent research data folders
7. preserve original data while creating standardized raw-data copies

Typical user requests:

- "把已有原始数据整理成带 metadata 的 HDF5"
- "帮我给仿真程序加上 HDF5 导出"
- "统一 raw data 命名"
- "把 csv/txt/dat/npy 结果整理成科研规范的数据格式"
- "给这个项目建立 raw data 规范"

---

## Scope Boundary

This skill is responsible for:

- raw-data format standardization
- raw-data naming
- metadata design
- HDF5 export code
- conversion from existing raw formats into HDF5
- validation of HDF5 structure

This skill is not responsible for:

- fitting models
- generating publication figures
- physical interpretation
- writing paper text
- modifying numerical algorithms unless required for data export
- deleting original raw files

Never delete or overwrite original raw data unless the user explicitly asks for that.

---

## Recommended Project Structure

For a research project, prefer:

```text
project/
├── raw/
│   ├── original/
│   │   ├── old_result_1.csv
│   │   └── old_result_2.txt
│   │
│   └── hdf5/
│       ├── 2026-05-30_90nmMOSFET_pulse_10K_1mW_run001.h5
│       └── 2026-05-30_FinFET_pulse_10K_1mW_run002.h5
│
├── processed/
│   ├── zth_curve.csv
│   └── peak_temperature.csv
│
├── scripts/
│   ├── convert_raw_to_hdf5.py
│   ├── export_hdf5.py
│   └── plot_zth.py
│
└── figures/
```

Principle:

```text
raw/original/      preserve original files
raw/hdf5/          standardized raw files
processed/         plotting-ready CSV files
figures/           PDF/SVG/PNG figure outputs
scripts/           conversion, processing, and plotting scripts
```

---

## Raw HDF5 File Naming Standard

Use this filename format:

```text
YYYY-MM-DD____runXXX.h5
```

Examples:

```text
2026-05-30_90nmMOSFET_pulse_10K_1mW_run001.h5
2026-05-30_FinFET_steady_10K_5mW_run002.h5
2026-05-30_GaNHEMT_zth_300K_10us_run003.h5
```

Rules:

- use ISO date: `YYYY-MM-DD`
- use compact device names: `90nmMOSFET`, `FinFET`, `GaNHEMT`
- use explicit test types: `pulse`, `steady`, `zth`, `iv`, `thermalBTE`, `spice`
- include one or two key conditions: `10K`, `1mW`, `10us`, `Vds1V`
- use zero-padded run index: `run001`, `run002`
- avoid spaces and special characters
- prefer ASCII characters for filenames

Do not use vague names such as:

```text
data.h5
test.h5
new_result.h5
final.h5
final_final.h5
```

---

## Required HDF5 Internal Structure

Each raw HDF5 file should follow this structure:

```text
/
├── data/
│   ├── time_s
│   ├── temperature_K
│   ├── power_W
│   ├── voltage_V
│   └── current_A
│
├── metadata/
│   └── attrs
│
└── attrs
```

Use `/data` for numerical arrays.

Use `/metadata` attributes for descriptive metadata.

Use root attributes for global format information.

---

## Dataset Naming Rules

Dataset names must be descriptive and include physical units when possible.

Good examples:

```text
time_s
time_us
temperature_K
delta_temperature_K
power_W
power_mW
current_A
voltage_V
thermal_resistance_K_per_W
x_m
y_m
z_m
heat_generation_W_per_m3
```

Bad examples:

```text
T
temp
x
y
data
result
array1
out
```

If unit is dimensionless, use a clear name:

```text
duty_cycle
iteration_index
normalized_temperature
```

---

## Required Metadata Fields

Each raw HDF5 file should include at least:

```yaml
experiment_id:
date:
source_file:
created_by:
software:
data_type:
device:
condition:
units:
processing_history:
notes:
```

For simulation data, also include when available:

```yaml
simulation_tool:
simulation_version:
mesh_info:
boundary_conditions:
material_model:
ambient_temperature_K:
power_W:
bias_condition:
geometry:
solver_settings:
```

For measurement data, also include when available:

```yaml
instrument:
operator:
calibration:
sampling_rate:
measurement_condition:
ambient_condition:
```

Do not silently invent uncertain metadata. If a field is unknown, use one of:

```text
unknown
not_provided
inferred_from_filename
```

When inferring metadata, record the inference in `processing_history`.

---

## Conversion Procedure for Existing Raw Data

When converting existing raw files:

1. Inspect filenames, folder structure, and column names.
2. Identify data arrays and physical units.
3. Preserve original raw files.
4. Create standardized `.h5` files under `raw/hdf5/`.
5. Store the original file path in `metadata/source_file`.
6. Store conversion date and conversion script in `metadata/processing_history`.
7. Use explicit dataset names with units.
8. Validate that the HDF5 file can be reopened and read.
9. Produce a short conversion report listing converted files and uncertain metadata.

Never delete original files unless the user explicitly requests it.

---

## Code Modification Procedure

When modifying existing Python programs:

1. Find all export points, such as:

```python
numpy.savetxt(...)
pandas.DataFrame.to_csv(...)
scipy.io.savemat(...)
np.save(...)
np.savez(...)
pickle.dump(...)
```

2. Add a reusable HDF5 export utility, preferably in:

```text
scripts/export_hdf5.py
```

3. Replace ad-hoc raw-data export with `export_raw_hdf5(...)`.
4. Keep CSV export only for processed plotting data.
5. Construct metadata near where simulation or measurement conditions are defined.
6. Generate filename using the standard naming helper.
7. Validate written HDF5 files after export.
8. Do not change the numerical algorithm unless necessary for export.

---

## Recommended Python Export Utility

When creating or updating Python code, use this pattern:

```python
from pathlib import Path
from datetime import datetime
import json
import re

import h5py
import numpy as np

def _safe_token(value: object) -> str:
    text = str(value).strip().replace(" ", "")
    text = re.sub(r"[^A-Za-z0-9_.+-]", "", text)
    return text or "unknown"

def make_raw_hdf5_name(date, device, test_type, key_condition, run_id):
    return (
        f"{_safe_token(date)}_"
        f"{_safe_token(device)}_"
        f"{_safe_token(test_type)}_"
        f"{_safe_token(key_condition)}_"
        f"run{int(run_id):03d}.h5"
    )

def export_raw_hdf5(output_path, data_dict, metadata):
    output_path = Path(output_path)
    output_path.parent.mkdir(parents=True, exist_ok=True)

    with h5py.File(output_path, "w") as f:
        data_group = f.create_group("data")
        metadata_group = f.create_group("metadata")

        for name, values in data_dict.items():
            arr = np.asarray(values)
            data_group.create_dataset(
                name,
                data=arr,
                compression="gzip",
                compression_opts=4,
            )

        for key, value in metadata.items():
            if isinstance(value, (dict, list, tuple)):
                metadata_group.attrs[key] = json.dumps(value, ensure_ascii=False)
            elif value is None:
                metadata_group.attrs[key] = "not_provided"
            else:
                metadata_group.attrs[key] = value

        f.attrs["created_at"] = datetime.now().isoformat(timespec="seconds")
        f.attrs["format"] = "raw-research-hdf5-v1"
        f.attrs["description"] = "Standardized raw research data with metadata"

    validate_raw_hdf5(output_path)
    return output_path

def validate_raw_hdf5(path):
    path = Path(path)
    if not path.exists():
        raise FileNotFoundError(path)

    with h5py.File(path, "r") as f:
        if "data" not in f:
            raise ValueError("Missing required group: /data")
        if "metadata" not in f:
            raise ValueError("Missing required group: /metadata")
        if len(f["data"].keys()) == 0:
            raise ValueError("/data contains no datasets")
        if f.attrs.get("format", "") != "raw-research-hdf5-v1":
            raise ValueError("Unexpected or missing HDF5 format attribute")

    return True
```

---

## Example Metadata for Device Thermal Simulation

```python
metadata = {
    "experiment_id": "sim_001",
    "date": "2026-05-30",
    "source_file": "simulation_output/original_result.csv",
    "created_by": "ChenKun Deng",
    "software": "Python",
    "data_type": "raw_simulation",
    "device": "90nmMOSFET",
    "condition": {
        "ambient_temperature_K": 10,
        "power_W": 1e-3,
        "bias": "not_provided",
    },
    "units": {
        "time_s": "s",
        "temperature_K": "K",
        "power_W": "W",
    },
    "simulation_tool": "phonon_BTE_solver",
    "mesh_info": "not_provided",
    "boundary_conditions": "not_provided",
    "material_model": "temperature-dependent phonon BTE",
    "processing_history": [
        "Converted to standardized raw HDF5 format."
    ],
    "notes": "Raw thermal simulation result."
}
```

---

## Example Data Export

```python
filename = make_raw_hdf5_name(
    date="2026-05-30",
    device="90nmMOSFET",
    test_type="pulse",
    key_condition="10K_1mW",
    run_id=1,
)

output_path = Path("raw/hdf5") / filename

data_dict = {
    "time_s": time,
    "temperature_K": temperature,
    "power_W": power,
}

export_raw_hdf5(output_path, data_dict, metadata)
```

---

## Processed Data Rule

Do not force processed plotting data into HDF5 unless the user explicitly wants that.

Preferred processed-data format:

```text
processed/*.csv
```

Processed CSV files should be easy for plotting tools to read.

Prefer tidy long-table format:

```csv
time_s,value,quantity,device,condition
1e-6,0.01,zth_K_per_W,90nmMOSFET,10K_1mW
2e-6,0.02,zth_K_per_W,90nmMOSFET,10K_1mW
```

---

## Validation Checklist

After creating or modifying HDF5 files, verify:

- filename follows the naming rule
- original raw data is preserved
- file opens successfully
- `/data` group exists
- `/metadata` group exists
- data arrays are stored under `/data`
- datasets have meaningful names
- units are included in dataset names or metadata
- source file is recorded
- date is recorded
- device or simulation target is recorded
- key condition is recorded
- processing history is recorded
- HDF5 format attribute is `raw-research-hdf5-v1`

---

## Reporting Format

After applying this skill, summarize:

```text
Created/updated:
- path/to/file.h5
- path/to/export_hdf5.py

Preserved originals:
- path/to/original.csv

Metadata uncertainty:
- ambient_temperature_K inferred from filename
- mesh_info not provided

Validation:
- all generated HDF5 files reopened successfully
```

---

## What Not to Do

Do not:

- overwrite original raw files
- delete original raw files
- mix raw and processed data without clear separation
- store plotting-only CSV data as raw data by default
- use vague dataset names such as `data1`, `result`, or `temp`
- silently invent metadata
- use pickle as a long-term scientific data format
- save final paper figures inside HDF5
- change simulation physics or numerical methods unless explicitly asked

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Narroog](https://github.com/Narroog)
- **Source:** [Narroog/research-skills](https://github.com/Narroog/research-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** yes
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-narroog-research-skills-raw-hdf5-standardization
- Seller: https://agentstack.voostack.com/s/narroog
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.