Raw Hdf5 Standardization

Raw HDF5 Standardization Skill

Purpose

This skill standardizes raw research data into metadata-rich HDF5 files and updates existing code so that future raw data is exported directly in the same format.

The goal is to make raw research data:

• traceable
• reproducible
• consistently named
• machine-readable
• suitable for long-term storage
• easy to convert into processed CSV files for plotting

This skill is mainly for raw data governance. It is not a plotting skill. Processed plotting data should usually be exported separately as CSV.

---

When to Use This Skill

Use this skill when the user asks to:

1. reorganize existing raw data files into HDF5 format
2. add metadata to raw data
3. rename raw data files according to a project convention
4. modify Python simulation or measurement code to export HDF5
5. create reusable raw-data export utilities
6. clean inconsistent research data folders
7. preserve original data while creating standardized raw-data copies

Typical user requests:

• "把已有原始数据整理成带 metadata 的 HDF5"
• "帮我给仿真程序加上 HDF5 导出"
• "统一 raw data 命名"
• "把 csv/txt/dat/npy 结果整理成科研规范的数据格式"
• "给这个项目建立 raw data 规范"

---

Scope Boundary

This skill is responsible for:

• raw-data format standardization
• raw-data naming
• metadata design
• HDF5 export code
• conversion from existing raw formats into HDF5
• validation of HDF5 structure

This skill is not responsible for:

• fitting models
• generating publication figures
• physical interpretation
• writing paper text
• modifying numerical algorithms unless required for data export
• deleting original raw files

Never delete or overwrite original raw data unless the user explicitly asks for that.

---

Recommended Project Structure

For a research project, prefer:

project/
├── raw/
│   ├── original/
│   │   ├── old_result_1.csv
│   │   └── old_result_2.txt
│   │
│   └── hdf5/
│       ├── 2026-05-30_90nmMOSFET_pulse_10K_1mW_run001.h5
│       └── 2026-05-30_FinFET_pulse_10K_1mW_run002.h5
│
├── processed/
│   ├── zth_curve.csv
│   └── peak_temperature.csv
│
├── scripts/
│   ├── convert_raw_to_hdf5.py
│   ├── export_hdf5.py
│   └── plot_zth.py
│
└── figures/

Principle:

raw/original/      preserve original files
raw/hdf5/          standardized raw files
processed/         plotting-ready CSV files
figures/           PDF/SVG/PNG figure outputs
scripts/           conversion, processing, and plotting scripts

---

Raw HDF5 File Naming Standard

Use this filename format:

YYYY-MM-DD____runXXX.h5

Examples:

2026-05-30_90nmMOSFET_pulse_10K_1mW_run001.h5
2026-05-30_FinFET_steady_10K_5mW_run002.h5
2026-05-30_GaNHEMT_zth_300K_10us_run003.h5

Rules:

• use ISO date: YYYY-MM-DD
• use compact device names: 90nmMOSFET, FinFET, GaNHEMT
• use explicit test types: pulse, steady, zth, iv, thermalBTE, spice
• include one or two key conditions: 10K, 1mW, 10us, Vds1V
• use zero-padded run index: run001, run002
• avoid spaces and special characters
• prefer ASCII characters for filenames

Do not use vague names such as:

data.h5
test.h5
new_result.h5
final.h5
final_final.h5

---

Required HDF5 Internal Structure

Each raw HDF5 file should follow this structure:

/
├── data/
│   ├── time_s
│   ├── temperature_K
│   ├── power_W
│   ├── voltage_V
│   └── current_A
│
├── metadata/
│   └── attrs
│
└── attrs

Use /data for numerical arrays.

Use /metadata attributes for descriptive metadata.

Use root attributes for global format information.

---

Dataset Naming Rules

Dataset names must be descriptive and include physical units when possible.

Good examples:

time_s
time_us
temperature_K
delta_temperature_K
power_W
power_mW
current_A
voltage_V
thermal_resistance_K_per_W
x_m
y_m
z_m
heat_generation_W_per_m3

Bad examples:

T
temp
x
y
data
result
array1
out

If unit is dimensionless, use a clear name:

duty_cycle
iteration_index
normalized_temperature

---

Required Metadata Fields

Each raw HDF5 file should include at least:

experiment_id:
date:
source_file:
created_by:
software:
data_type:
device:
condition:
units:
processing_history:
notes:

For simulation data, also include when available:

simulation_tool:
simulation_version:
mesh_info:
boundary_conditions:
material_model:
ambient_temperature_K:
power_W:
bias_condition:
geometry:
solver_settings:

For measurement data, also include when available:

instrument:
operator:
calibration:
sampling_rate:
measurement_condition:
ambient_condition:

Do not silently invent uncertain metadata. If a field is unknown, use one of:

unknown
not_provided
inferred_from_filename

When inferring metadata, record the inference in processing_history.

---

Conversion Procedure for Existing Raw Data

When converting existing raw files:

1. Inspect filenames, folder structure, and column names.
2. Identify data arrays and physical units.
3. Preserve original raw files.
4. Create standardized .h5 files under raw/hdf5/.
5. Store the original file path in metadata/source_file.
6. Store conversion date and conversion script in metadata/processing_history.
7. Use explicit dataset names with units.
8. Validate that the HDF5 file can be reopened and read.
9. Produce a short conversion report listing converted files and uncertain metadata.

Never delete original files unless the user explicitly requests it.

---

Code Modification Procedure

When modifying existing Python programs:

1. Find all export points, such as:

numpy.savetxt(...)
pandas.DataFrame.to_csv(...)
scipy.io.savemat(...)
np.save(...)
np.savez(...)
pickle.dump(...)

1. Add a reusable HDF5 export utility, preferably in:

scripts/export_hdf5.py

1. Replace ad-hoc raw-data export with export_raw_hdf5(...).
2. Keep CSV export only for processed plotting data.
3. Construct metadata near where simulation or measurement conditions are defined.
4. Generate filename using the standard naming helper.
5. Validate written HDF5 files after export.
6. Do not change the numerical algorithm unless necessary for export.

---

Recommended Python Export Utility

When creating or updating Python code, use this pattern:

from pathlib import Path
from datetime import datetime
import json
import re

import h5py
import numpy as np

def _safe_token(value: object) -> str:
    text = str(value).strip().replace(" ", "")
    text = re.sub(r"[^A-Za-z0-9_.+-]", "", text)
    return text or "unknown"

def make_raw_hdf5_name(date, device, test_type, key_condition, run_id):
    return (
        f"{_safe_token(date)}_"
        f"{_safe_token(device)}_"
        f"{_safe_token(test_type)}_"
        f"{_safe_token(key_condition)}_"
        f"run{int(run_id):03d}.h5"
    )

def export_raw_hdf5(output_path, data_dict, metadata):
    output_path = Path(output_path)
    output_path.parent.mkdir(parents=True, exist_ok=True)

    with h5py.File(output_path, "w") as f:
        data_group = f.create_group("data")
        metadata_group = f.create_group("metadata")

        for name, values in data_dict.items():
            arr = np.asarray(values)
            data_group.create_dataset(
                name,
                data=arr,
                compression="gzip",
                compression_opts=4,
            )

        for key, value in metadata.items():
            if isinstance(value, (dict, list, tuple)):
                metadata_group.attrs[key] = json.dumps(value, ensure_ascii=False)
            elif value is None:
                metadata_group.attrs[key] = "not_provided"
            else:
                metadata_group.attrs[key] = value

        f.attrs["created_at"] = datetime.now().isoformat(timespec="seconds")
        f.attrs["format"] = "raw-research-hdf5-v1"
        f.attrs["description"] = "Standardized raw research data with metadata"

    validate_raw_hdf5(output_path)
    return output_path

def validate_raw_hdf5(path):
    path = Path(path)
    if not path.exists():
        raise FileNotFoundError(path)

    with h5py.File(path, "r") as f:
        if "data" not in f:
            raise ValueError("Missing required group: /data")
        if "metadata" not in f:
            raise ValueError("Missing required group: /metadata")
        if len(f["data"].keys()) == 0:
            raise ValueError("/data contains no datasets")
        if f.attrs.get("format", "") != "raw-research-hdf5-v1":
            raise ValueError("Unexpected or missing HDF5 format attribute")

    return True

---

Example Metadata for Device Thermal Simulation

metadata = {
    "experiment_id": "sim_001",
    "date": "2026-05-30",
    "source_file": "simulation_output/original_result.csv",
    "created_by": "ChenKun Deng",
    "software": "Python",
    "data_type": "raw_simulation",
    "device": "90nmMOSFET",
    "condition": {
        "ambient_temperature_K": 10,
        "power_W": 1e-3,
        "bias": "not_provided",
    },
    "units": {
        "time_s": "s",
        "temperature_K": "K",
        "power_W": "W",
    },
    "simulation_tool": "phonon_BTE_solver",
    "mesh_info": "not_provided",
    "boundary_conditions": "not_provided",
    "material_model": "temperature-dependent phonon BTE",
    "processing_history": [
        "Converted to standardized raw HDF5 format."
    ],
    "notes": "Raw thermal simulation result."
}

---

Example Data Export

filename = make_raw_hdf5_name(
    date="2026-05-30",
    device="90nmMOSFET",
    test_type="pulse",
    key_condition="10K_1mW",
    run_id=1,
)

output_path = Path("raw/hdf5") / filename

data_dict = {
    "time_s": time,
    "temperature_K": temperature,
    "power_W": power,
}

export_raw_hdf5(output_path, data_dict, metadata)

---

Processed Data Rule

Do not force processed plotting data into HDF5 unless the user explicitly wants that.

Preferred processed-data format:

processed/*.csv

Processed CSV files should be easy for plotting tools to read.

Prefer tidy long-table format:

time_s,value,quantity,device,condition
1e-6,0.01,zth_K_per_W,90nmMOSFET,10K_1mW
2e-6,0.02,zth_K_per_W,90nmMOSFET,10K_1mW

---

Validation Checklist

After creating or modifying HDF5 files, verify:

• filename follows the naming rule
• original raw data is preserved
• file opens successfully
• /data group exists
• /metadata group exists
• data arrays are stored under /data
• datasets have meaningful names
• units are included in dataset names or metadata
• source file is recorded
• date is recorded
• device or simulation target is recorded
• key condition is recorded
• processing history is recorded
• HDF5 format attribute is raw-research-hdf5-v1

---

Reporting Format

After applying this skill, summarize:

Created/updated:
- path/to/file.h5
- path/to/export_hdf5.py

Preserved originals:
- path/to/original.csv

Metadata uncertainty:
- ambient_temperature_K inferred from filename
- mesh_info not provided

Validation:
- all generated HDF5 files reopened successfully

---

What Not to Do

Do not:

• overwrite original raw files
• delete original raw files
• mix raw and processed data without clear separation
• store plotting-only CSV data as raw data by default
• use vague dataset names such as data1, result, or temp
• silently invent metadata
• use pickle as a long-term scientific data format
• save final paper figures inside HDF5
• change simulation physics or numerical methods unless explicitly asked

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Narroog
• Source: Narroog/research-skills
• License: MIT

Install and usage instructions live in the source repository linked above.