# Ml Model Evaluation > Evaluate machine learning models rigorously — with train/test splits, cross-validation, business metric alignment, bias detection, and production readiness assessment. - **Type:** Skill - **Install:** `agentstack add skill-the-ai-directory-company-agents-and-skills-ml-model-evaluation` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [The-AI-Directory-Company](https://agentstack.voostack.com/s/the-ai-directory-company) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [The-AI-Directory-Company](https://github.com/The-AI-Directory-Company) - **Source:** https://github.com/The-AI-Directory-Company/agents-and-skills/tree/main/skills/ml-model-evaluation - **Website:** https://ai-directory.company ## Install ```sh agentstack add skill-the-ai-directory-company-agents-and-skills-ml-model-evaluation ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # ML Model Evaluation ## Before you start Gather the following from the user. If anything is missing, ask before proceeding: 1. **What problem is the model solving?** — Classification, regression, ranking, recommendation, generation 2. **What is the business objective?** — The real-world outcome (reduce churn, detect fraud, recommend products) 3. **What data is available?** — Dataset size, feature count, label quality, class balance, time range 4. **What are the constraints?** — Latency, model size, interpretability needs, regulatory obligations 5. **What is the baseline?** — Current system performance (rule-based, human, or previous model) 6. **What is the cost of errors?** — False positive vs false negative impact in business terms ## Evaluation template ### 1. Define Success Metrics Map business objectives to technical metrics. Never evaluate on technical metrics alone. ``` Business Objective: Detect fraudulent transactions before settlement Primary Metric: Precision at 95% recall Secondary Metrics: AUC-ROC, F1 score, false positive rate Business Constraint: 85% precision at 95% recall ``` Metric selection rules: - **Classification**: Use precision/recall/F1 for imbalanced classes. Accuracy is misleading when 98% of data is one class. - **Regression**: MAE for outlier-tolerant, RMSE when large errors are disproportionately costly. - **Ranking**: NDCG/MAP when order matters, precision@k when only top results matter. - Always include a business metric: revenue impact, time saved, error cost reduction. ### 2. Data Splitting Strategy **Random split** — Default for i.i.d. data: Train 70% / Validation 15% / Test 15%. **Temporal split** — Required for time-dependent data: Train before T1 / Validation T1-T2 / Test after T2. **Stratified split** — Required for imbalanced classification: maintain class proportions across splits. **Group split** — Required when one entity has multiple samples: split by entity ID, not by row. Critical rules: - Never use test data for any decision — tuning, feature selection, or threshold setting - For small datasets (10% across segments, investigate. - **Error distribution**: For regression, plot residuals — are errors uniform or concentrated? ### 5. Bias Detection Check for disparities across protected groups: - **Demographic parity**: Does positive prediction rate differ across groups? - **Equal opportunity**: Does true positive rate differ across groups? - **Calibration**: Does a predicted 80% probability mean 80% actual positive rate for all groups? If disparities exceed acceptable thresholds, investigate data representation, feature encoding, and model architecture. ### 6. Production Readiness Verify before deployment: meets primary metric target, meets latency constraint, model size within limits, bias assessment passed, monitoring plan defined (prediction drift, feature drift, business metric tracking), fallback strategy documented, A/B test plan prepared, data pipeline validated, model versioning in place. ## Quality checklist Before delivering a model evaluation, verify: - [ ] Business objective is mapped to technical metrics with a stated target - [ ] Data split strategy matches data characteristics (temporal, imbalanced, grouped) - [ ] Test set was never used for model selection or tuning - [ ] At least one simple baseline is included for comparison - [ ] Error analysis examines specific failure cases, not just aggregates - [ ] Performance is broken down by relevant segments - [ ] Bias detection covers protected attributes and business segments - [ ] Production readiness includes latency, monitoring, and fallback ## Common mistakes - **Evaluating on accuracy alone.** A model predicting "not fraud" for everything achieves 99.5% accuracy on a 0.5% fraud dataset. Use precision/recall for imbalanced problems. - **Leaking test data.** Using the test set for feature selection or tuning inflates results and breaks the generalization guarantee. - **Ignoring the simple baseline.** A logistic regression at 90% in 30 seconds often beats a deep learning model at 92% after two weeks of engineering. - **Reporting only aggregate metrics.** 90% overall accuracy that drops to 50% on a critical segment is not a 90%-accurate model for those users. - **Skipping the cost analysis.** False positives and false negatives rarely cost the same. The evaluation must reflect the asymmetry. - **No production monitoring plan.** Models degrade as distributions shift. An evaluation without monitoring is incomplete. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [The-AI-Directory-Company](https://github.com/The-AI-Directory-Company) - **Source:** [The-AI-Directory-Company/agents-and-skills](https://github.com/The-AI-Directory-Company/agents-and-skills) - **License:** MIT - **Homepage:** https://ai-directory.company Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-the-ai-directory-company-agents-and-skills-ml-model-evaluation - Seller: https://agentstack.voostack.com/s/the-ai-directory-company - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.