Imbalanced Data

Imbalanced Data

Overview

When positives are rare, naive training and naive metrics both mislead. A model that always predicts "negative" can score 99% accuracy and catch zero fraud. Handle imbalance at three levels: metric, algorithm, and threshold — and resample inside cross-validation, never before.

When to use

• Class ratio is skewed (e.g., 95/5 or worse).
• Catching the rare class matters (fraud, defaults, rare disease, defects).

Step 1 — Fix the metric first

Drop accuracy and ROC-AUC-as-sole-metric. Prefer:

• PR-AUC (average precision) — most informative for rare positives.
• Recall @ fixed precision — "catch X% of fraud while keeping false alarms tolerable."
• F1 / Fβ — β>1 weights recall when misses are costly.

Step 2 — Algorithm-level handling

| Technique | How | When | |-----------|-----|------| | Class weights | class_weight="balanced" / scale_pos_weight | First choice — no data duplication | | Undersample majority | RandomUnderSampler | Lots of data, majority redundant | | Oversample minority | SMOTE / ADASYN | Limited minority samples | | Combine | SMOTE + Tomek/ENN | Noisy boundaries |

Class weights are the cheapest, leak-free first move:

# sklearn
LogisticRegression(class_weight="balanced")
# XGBoost / LightGBM
scale_pos_weight = n_negative / n_positive

Step 3 — Resample INSIDE the pipeline (no leakage)

Resampling before CV leaks synthetic neighbors across the split and inflates scores. Use imblearn's pipeline so SMOTE fits on training folds only:

from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold

pipe = ImbPipeline([
    ("smote", SMOTE(random_state=42)),   # applied to train fold only
    ("clf", HistGradientBoostingClassifier(random_state=42)),
])
cv = StratifiedKFold(5, shuffle=True, random_state=42)
print(cross_val_score(pipe, X, y, cv=cv, scoring="average_precision").mean())

Step 4 — Tune the decision threshold

Default 0.5 is almost always wrong for imbalanced problems. Pick the threshold from the validation PR curve to hit your precision/recall target:

from sklearn.metrics import precision_recall_curve
import numpy as np

prec, rec, thr = precision_recall_curve(y_val, val_scores)
# smallest threshold achieving >= 0.90 precision
ok = np.where(prec[:-1] >= 0.90)[0]
chosen = thr[ok[0]] if len(ok) else 0.5

Pitfalls

• Accuracy as the metric — the central trap.
• SMOTE before train/test split — leaks synthetic points; scores look great, production fails.
• Resampling the test set — evaluate on the real (imbalanced) distribution.
• Oversampling then ignoring the threshold — probabilities shift; recalibrate/tune the cut.
• Synthetic samples on categorical/high-dim data — SMOTE assumes meaningful interpolation; use SMOTENC or class weights instead.

Hand-off

A model trained with leak-free resampling/weights, scored with PR-AUC and a tuned threshold via model-evaluation, ready for model-serving.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: param087
• Source: param087/agent-ml-skills
• License: MIT

Install and usage instructions live in the source repository linked above.