# Hierarchical Taxonomy Clustering

> Build unified multi-level category taxonomy from hierarchical product category paths from any e-commerce companies using embedding-based recursive clustering with intelligent category naming via weighted word frequency analysis.

- **Type:** Skill
- **Install:** `agentstack add skill-benchflow-ai-skillsbench-hierarchical-taxonomy-clustering`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [benchflow-ai](https://agentstack.voostack.com/s/benchflow-ai)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [benchflow-ai](https://github.com/benchflow-ai)
- **Source:** https://github.com/benchflow-ai/skillsbench/tree/main/tasks-extra/taxonomy-tree-merge/environment/skills/hierarchical-taxonomy-clustering
- **Website:** https://www.skillsbench.ai

## Install

```sh
agentstack add skill-benchflow-ai-skillsbench-hierarchical-taxonomy-clustering
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Hierarchical Taxonomy Clustering

Create a unified multi-level taxonomy from hierarchical category paths by clustering similar paths and automatically generating meaningful category names.

## Problem

Given category paths from multiple sources (e.g., "electronics -> computers -> laptops"), create a unified taxonomy that groups similar paths across sources, generates meaningful category names, and produces a clean N-level hierarchy (typically 5 levels). The unified category taxonomy could be used to do analysis or metric tracking on products from different platform.

## Methodology

1. **Hierarchical Weighting**: Convert paths to embeddings with exponentially decaying weights (Level i gets weight 0.6^(i-1)) to signify the importance of category granularity
2. **Recursive Clustering**: Hierarchically cluster at each level (10-20 clusters at L1, 3-20 at L2-L5) using cosine distance
3. **Intelligent Naming**: Generate category names via weighted word frequency + lemmatization + bundle word logic
4. **Quality Control**: Exclude all ancestor words (parent, grandparent, etc.), avoid ancestor path duplicates, clean special characters

## Output

DataFrame with added columns:
- `unified_level_1`: Top-level category (e.g., "electronic | device")
- `unified_level_2`: Second-level category (e.g., "computer | laptop")
- `unified_level_3` through `unified_level_N`: Deeper levels

Category names use ` | ` separator, max 5 words, covering 70%+ of records in each cluster.

## Installation

```bash
pip install pandas numpy scipy sentence-transformers nltk tqdm
python -c "import nltk; nltk.download('wordnet'); nltk.download('omw-1.4')"
```

## 4-Step Pipeline

### Step 1: Load, Standardize, Filter and Merge (`step1_preprocessing_and_merge.py`)
- **Input**: List of (DataFrame, source_name) tuples, each of the  with `category_path` column
- **Process**: Per-source deduplication, text cleaning (remove &/,/'/-/quotes,'and' or "&", "," and so on, lemmatize words as nouns), normalize delimiter to ` > `, depth filtering, prefix removal, then merge all sources. source_level should reflect the processed version of the source level name
- **Output**: Merged DataFrame with `category_path`, `source`, `depth`, `source_level_1` through `source_level_N`

### Step 2: Weighted Embeddings (`step2_weighted_embedding_generation.py`)
- **Input**: DataFrame from Step 1
- **Output**: Numpy embedding matrix (n_records × 384)
- Weights: L1=1.0, L2=0.6, L3=0.36, L4=0.216, L5=0.1296 (exponential decay 0.6^(n-1))
- **Performance**: For ~10,000 records, expect 2-5 minutes. Progress bar will show encoding status.

### Step 3: Recursive Clustering (`step3_recursive_clustering_naming.py`)
- **Input**: DataFrame + embeddings from Step 2
- **Output**: Assignments dict {index → {level_1: ..., level_5: ...}}
- Average linkage + cosine distance, 10-20 clusters at L1, 3-20 at L2-L5
- Word-based naming: weighted frequency + lemmatization + coverage ≥70%
- **Performance**: For ~10,000 records, expect 1-3 minutes for hierarchical clustering and naming. Be patient - the system is working through recursive levels.

### Step 4: Export Results (`step4_result_assignments.py`)
- **Input**: DataFrame + assignments from Step 3
- **Output**:
  - `unified_taxonomy_full.csv` - all records with unified categories
  - `unified_taxonomy_hierarchy.csv` - unique taxonomy structure

## Usage

**Use `scripts/pipeline.py` to run the complete 4-step workflow.**

See `scripts/pipeline.py` for:
- Complete implementation of all 4 steps
- Example code for processing multiple sources
- Command-line interface
- Individual step usage (for advanced control)

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [benchflow-ai](https://github.com/benchflow-ai)
- **Source:** [benchflow-ai/skillsbench](https://github.com/benchflow-ai/skillsbench)
- **License:** Apache-2.0
- **Homepage:** https://www.skillsbench.ai

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-benchflow-ai-skillsbench-hierarchical-taxonomy-clustering
- Seller: https://agentstack.voostack.com/s/benchflow-ai
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.