# Data Pipeline

> ETL and data transformation: CSV, JSON, Parquet, NDJSON processing with filtering, mapping, joining, aggregating.

- **Type:** Skill
- **Install:** `agentstack add skill-jansenanalytics-claudex-data-pipeline`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [JansenAnalytics](https://agentstack.voostack.com/s/jansenanalytics)
- **Installs:** 0
- **Category:** [Data & Analytics](https://agentstack.voostack.com/c/data-and-analytics)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [JansenAnalytics](https://github.com/JansenAnalytics)
- **Source:** https://github.com/JansenAnalytics/claudex/tree/main/skills/data-pipeline

## Install

```sh
agentstack add skill-jansenanalytics-claudex-data-pipeline
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# data-pipeline

ETL and data transformation: CSV, JSON, Parquet, NDJSON processing with filtering, mapping, joining, aggregating.

## When to Use

- Converting between data formats (CSV ↔ JSON, NDJSON, etc.)
- Cleaning or filtering datasets
- Merging/joining files on a common key
- Aggregating data (sum, avg, count, min, max by group)
- Detecting file formats automatically
- Building multi-step data processing pipelines

## Scripts

| Script             | Purpose                  | Usage                                                                 |
| ------------------ | ------------------------ | --------------------------------------------------------------------- |
| `csv-to-json.py`   | Convert CSV → JSON array | `python3 csv-to-json.py input.csv [output.json]`                      |
| `json-to-csv.py`   | Convert JSON array → CSV | `python3 json-to-csv.py input.json [output.csv]`                      |
| `data-merge.py`    | Join two files on a key  | `python3 data-merge.py left.csv right.csv --key id [--how inner]`     |
| `data-filter.py`   | Filter + aggregate data  | `python3 data-filter.py input.csv --filter "col=val" --agg "col:sum"` |
| `detect-format.py` | Auto-detect file format  | `python3 detect-format.py file.ext`                                   |

All scripts read from stdin if no file argument given. Output goes to stdout by default.

## References

- `references/jq-cheatsheet.md` — Common jq patterns for JSON processing
- `references/csvkit-patterns.md` — csvkit one-liners
- `references/large-file-handling.md` — Strategies for big datasets

## Examples

```bash
# Convert CSV to JSON
python3 scripts/csv-to-json.py sales.csv > sales.json

# Filter rows and aggregate
python3 scripts/data-filter.py sales.csv --filter "region=Europe" --agg "revenue:sum,count"

# Merge two CSVs on 'id' column
python3 scripts/data-merge.py users.csv orders.csv --key user_id --how left

# Detect format
python3 scripts/detect-format.py mystery_file.dat
```

## Dependencies

- Python 3 (standard library only — csv, json, argparse)
- No external packages required

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [JansenAnalytics](https://github.com/JansenAnalytics)
- **Source:** [JansenAnalytics/claudex](https://github.com/JansenAnalytics/claudex)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-jansenanalytics-claudex-data-pipeline
- Seller: https://agentstack.voostack.com/s/jansenanalytics
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.