Super Ai Ml Ops

Super AI/ML Ops

Overview

Make AI systems stable and measurable in production.

User Intent Examples

• "Need help with LLM Evaluation for my product/site."
• "Create a plan for LLM Ops."
• "Not sure where to start, need a quick assessment."

Workflow

1. Define evaluation metrics, datasets, and acceptance thresholds.
2. Set up observability for quality, latency, and errors.
3. Implement caching and cost controls.
4. Create monitoring and alerting for regressions.
5. Establish release and rollback procedures for prompts/models.
6. Document runbooks and ongoing QA cadence.

Minimal Intake Questions

• Primary goal or outcome
• Scope (pages, systems, teams, or timeframe)
• Constraints (tools, budget, timeline)

Output Format

• Eval plan and scoring rubric
• Monitoring and alerting checklist
• Cost and caching strategy
• Release and rollback plan
• Runbook and QA cadence

Routing Map (Modules)

• LLM Evaluation -> references/modules/llm-evaluation.md
• LLM Ops -> references/modules/llm-ops.md

Bundled References

• references/modules/
• scripts/
• assets/
• agents/

Compatibility Notes

• If any module references slash commands or tool-specific paths, translate them into plain-language steps.
• Keep outputs platform-agnostic unless the user specifies a specific tool, stack, or agent.

Guardrails

• Do not rely on single metrics; include qualitative checks.
• Track cost per request and cap budgets.
• Treat prompt/model updates as production changes.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: arpitexplores
• Source: arpitexplores/skills-super
• License: MIT

Install and usage instructions live in the source repository linked above.