Implementing Mlops

MLOps Patterns

Operationalize machine learning models from experimentation to production deployment and monitoring.

Purpose

Provide strategic guidance for ML engineers and platform teams to build production-grade ML infrastructure. Cover the complete lifecycle: experiment tracking, model registry, feature stores, deployment patterns, pipeline orchestration, and monitoring.

When to Use This Skill

Use this skill when:

• Designing MLOps infrastructure for production ML systems
• Selecting experiment tracking platforms (MLflow, Weights & Biases, Neptune)
• Implementing feature stores for online/offline feature serving
• Choosing model serving solutions (Seldon Core, KServe, BentoML, TorchServe)
• Building ML pipelines for training, evaluation, and deployment
• Setting up model monitoring and drift detection
• Establishing model governance and compliance frameworks
• Optimizing ML inference costs and performance
• Migrating from notebooks to production ML systems
• Implementing continuous training and automated retraining

Core Concepts

1. Experiment Tracking

Track experiments systematically to ensure reproducibility and collaboration.

Key Components:

• Parameters: Hyperparameters logged for each training run
• Metrics: Performance measures tracked over time (accuracy, loss, F1)
• Artifacts: Model weights, plots, datasets, configuration files
• Metadata: Tags, descriptions, Git commit SHA, environment details

Platform Comparison:

MLflow (Open-source standard):

• Framework-agnostic (PyTorch, TensorFlow, scikit-learn, XGBoost)
• Self-hosted or cloud-agnostic deployment
• Integrated model registry
• Basic UI, adequate for most use cases
• Free, requires infrastructure management

Weights & Biases (SaaS, collaboration-focused):

• Advanced visualization and dashboards
• Integrated hyperparameter optimization (Sweeps)
• Excellent team collaboration features
• SaaS pricing scales with usage
• Best-in-class UI

Neptune.ai (Enterprise-grade):

• Enterprise features (RBAC, audit logs, compliance)
• Integrated production monitoring
• Higher cost than W&B
• Good for regulated industries

Selection Criteria:

• Open-source requirement → MLflow
• Team collaboration critical → Weights & Biases
• Enterprise compliance (RBAC, audits) → Neptune.ai
• Hyperparameter optimization primary → Weights & Biases (Sweeps)

For detailed comparison and decision framework, see [references/experiment-tracking.md](references/experiment-tracking.md).

2. Model Registry and Versioning

Centralize model artifacts with version control and stage management.

Model Registry Components:

• Model artifacts (weights, serialized models)
• Training metrics (accuracy, F1, AUC)
• Hyperparameters used during training
• Training dataset version
• Feature schema (input/output signatures)
• Model cards (documentation, use cases, limitations)

Stage Management:

• None: Newly registered model
• Staging: Testing in pre-production environment
• Production: Serving live traffic
• Archived: Deprecated, retained for compliance

Versioning Strategies:

Semantic Versioning for Models:

• Major version (v2.0.0): Breaking change in input/output schema
• Minor version (v1.1.0): New feature, backward-compatible
• Patch version (v1.0.1): Bug fix, model retrained on new data

Git-Based Versioning:

• Model code in Git (training scripts, configuration)
• Model weights in DVC (Data Version Control) or Git-LFS
• Reproducibility via commit SHA + data version hash

For model lineage tracking and registry patterns, see [references/model-registry.md](references/model-registry.md).

3. Feature Stores

Centralize feature engineering to ensure consistency between training and inference.

Problem Addressed: Training/serving skew

• Training: Features computed with future knowledge (data leakage)
• Inference: Features computed with only past data
• Result: Model performs well in training but fails in production

Feature Store Solution:

Online Feature Store:

• Purpose: Low-latency feature retrieval for real-time inference
• Storage: Redis, DynamoDB, Cassandra (key-value stores)
• Latency: Sub-10ms for feature lookup
• Use Case: Real-time predictions (fraud detection, recommendations)

Offline Feature Store:

• Purpose: Historical feature data for training and batch inference
• Storage: Parquet files (S3/GCS), data warehouses (Snowflake, BigQuery)
• Latency: Seconds to minutes (batch retrieval)
• Use Case: Model training, backtesting, batch predictions

Point-in-Time Correctness:

• Ensures no future data leakage during training
• Feature values at time T only use data available before time T
• Critical for avoiding overly optimistic training metrics

Platform Comparison:

Feast (Open-source, cloud-agnostic):

• Most popular open-source feature store
• Supports Redis, DynamoDB, Datastore (online) and Parquet, BigQuery, Snowflake (offline)
• Cloud-agnostic, no vendor lock-in
• Active community, growing adoption

Tecton (Managed, production-grade):

• Feast-compatible API
• Fully managed service
• Integrated monitoring and governance
• Higher cost, enterprise-focused

SageMaker Feature Store (AWS):

• Integrated with AWS ecosystem
• Managed online/offline stores
• AWS lock-in

Databricks Feature Store (Databricks):

• Unity Catalog integration
• Delta Lake for offline storage
• Databricks ecosystem lock-in

Selection Criteria:

• Open-source, cloud-agnostic → Feast
• Managed solution, production-grade → Tecton
• AWS ecosystem → SageMaker Feature Store
• Databricks users → Databricks Feature Store

For feature engineering patterns and implementation, see [references/feature-stores.md](references/feature-stores.md).

4. Model Serving Patterns

Deploy models for synchronous, asynchronous, batch, or streaming inference.

Serving Patterns:

REST API Deployment:

• Pattern: HTTP endpoint for synchronous predictions
• Latency: 100ms → Alert
• Error rate > 1% → Alert
• Accuracy drop > 5% → Trigger retraining

Business Metrics Monitoring:

• Downstream impact: Conversion rate, revenue, user satisfaction
• Model predictions → business outcomes correlation
• Use Case: Optimize models for business value, not just ML metrics

Tools:

• Evidently AI: Data drift, model drift, data quality reports
• Prometheus + Grafana: Performance metrics, custom dashboards
• Arize AI: ML observability platform
• Fiddler: Model monitoring and explainability

For monitoring architecture and implementation, see [references/model-monitoring.md](references/model-monitoring.md).

8. Model Optimization Techniques

Reduce model size and inference latency.

Quantization:

• Convert model weights from float32 to int8
• Model size reduction: 4x smaller
• Inference speed: 2-3x faster
• Accuracy impact: Minimal (500 people): Neptune.ai (compliance) or MLflow (cost)

For detailed decision framework, see [references/decision-frameworks.md](references/decision-frameworks.md#experiment-tracking).

Framework 2: Feature Store Selection

Decision Matrix:

Primary requirement:

• Open-source, cloud-agnostic → Feast
• Managed solution, production-grade, multi-cloud → Tecton
• AWS ecosystem → SageMaker Feature Store
• GCP ecosystem → Vertex AI Feature Store
• Azure ecosystem → Azure ML Feature Store
• Databricks users → Databricks Feature Store
• Self-hosted with UI → Hopsworks

Criteria Comparison:

| Factor | Feast | Tecton | Hopsworks | SageMaker FS | |--------|-------|--------|-----------|--------------| | Cost | Free | $$$$ | Free (self-host) | $$$ | | Online Serving | Redis, DynamoDB | Managed | RonDB | Managed | | Offline Store | Parquet, BigQuery, Snowflake | Managed | Hive, S3 | S3 | | Point-in-Time | Yes | Yes | Yes | Yes | | Monitoring | External | Integrated | Basic | External | | Cloud Lock-in | No | No | No | AWS |

Recommendation:

• Open-source, self-managed → Feast
• Managed, production-grade → Tecton
• AWS ecosystem → SageMaker Feature Store
• Databricks users → Databricks Feature Store

For detailed decision framework, see [references/decision-frameworks.md](references/decision-frameworks.md#feature-store).

Framework 3: Model Serving Platform Selection

Decision Tree:

Infrastructure:

• Kubernetes-based → Advanced deployment patterns needed?
• Yes → Seldon Core (most features) or KServe (CNCF standard)
• No → BentoML (simpler, Python-first)
• Cloud-native (managed) → Cloud provider?
• AWS → SageMaker Endpoints
• GCP → Vertex AI Endpoints
• Azure → Azure ML Endpoints
• Framework-specific → Framework?
• PyTorch → TorchServe
• TensorFlow → TensorFlow Serving
• Serverless / minimal infrastructure → BentoML or Cloud Functions

Detailed Criteria:

| Feature | Seldon Core | KServe | BentoML | TorchServe | |---------|-------------|--------|---------|------------| | Kubernetes-Native | Yes | Yes | Optional | No | | Multi-Framework | Yes | Yes | Yes | PyTorch-only | | Deployment Strategies | Excellent | Good | Basic | Basic | | Explainability | Integrated | Integrated | External | No | | Complexity | High | Medium | Low | Low | | Learning Curve | Steep | Medium | Gentle | Gentle |

Recommendation:

• Kubernetes, advanced deployments → Seldon Core or KServe
• Python-first, simplicity → BentoML
• PyTorch-specific → TorchServe
• TensorFlow-specific → TensorFlow Serving
• Managed solution → SageMaker/Vertex AI/Azure ML

For detailed decision framework, see [references/decision-frameworks.md](references/decision-frameworks.md#model-serving).

Framework 4: ML Pipeline Orchestration Selection

Decision Matrix:

Primary use case:

• ML-specific pipelines, Kubernetes-native → Kubeflow Pipelines
• General-purpose orchestration, mature ecosystem → Apache Airflow
• Data science workflows, ease of use → Metaflow
• Modern approach, asset-based thinking → Dagster
• Dynamic workflows, Python-native → Prefect

Criteria Comparison:

| Factor | Kubeflow | Airflow | Metaflow | Dagster | Prefect | |--------|----------|---------|----------|---------|---------| | ML-Specific | Excellent | Good | Excellent | Good | Good | | Kubernetes | Native | Compatible | Optional | Compatible | Compatible | | Learning Curve | Steep | Steep | Gentle | Medium | Medium | | Maturity | High | Very High | Medium | Medium | Medium | | Community | Large | Very Large | Growing | Growing | Growing |

Recommendation:

• ML-specific, Kubernetes → Kubeflow Pipelines
• Mature, battle-tested → Apache Airflow
• Data scientists → Metaflow
• Software engineers → Dagster
• Modern, simpler than Airflow → Prefect

For detailed decision framework, see [references/decision-frameworks.md](references/decision-frameworks.md#orchestration).

Implementation Patterns

Pattern 1: End-to-End ML Pipeline

Automate the complete ML workflow from data to deployment.

Pipeline Stages:

1. Data Validation (Great Expectations)
2. Feature Engineering (transform raw data)
3. Data Splitting (train/validation/test)
4. Model Training (with hyperparameter tuning)
5. Model Evaluation (accuracy, fairness, explainability)
6. Model Registration (push to MLflow registry)
7. Deployment (promote to staging/production)

Architecture:

Data Lake → Data Validation → Feature Engineering → Training → Evaluation
    ↓
Model Registry (staging) → Testing → Production Deployment

For implementation details and code examples, see [references/ml-pipelines.md](references/ml-pipelines.md#end-to-end).

Pattern 2: Continuous Training

Automate model retraining based on drift detection.

Workflow:

1. Monitor production data for distribution changes
2. Detect data drift (KS test, PSI)
3. Trigger automated retraining pipeline
4. Validate new model (accuracy, fairness)
5. Deploy via canary strategy (5% → 100%)
6. Monitor new model performance
7. Rollback if metrics degrade

Trigger Conditions:

• Scheduled: Daily/weekly retraining
• Data drift: KS test p-value 5%
• Data volume: New training data exceeds threshold (10K samples)

For implementation details, see [references/ml-pipelines.md](references/ml-pipelines.md#continuous-training).

Pattern 3: Feature Store Integration

Ensure consistent features between training and inference.

Architecture:

Offline Store (Training):
  Parquet/BigQuery → Point-in-Time Join → Training Dataset

Online Store (Inference):
  Redis/DynamoDB → Low-Latency Lookup → Real-Time Prediction

Point-in-Time Correctness:

• Training: Fetch features as of specific timestamps (no future data)
• Inference: Fetch latest features (only past data)
• Guarantee: Same feature logic in training and inference

For implementation details and code examples, see [references/feature-stores.md](references/feature-stores.md#integration).

Pattern 4: Shadow Deployment Testing

Test new models in production without risk.

Workflow:

1. Deploy new model (v2) in shadow mode
2. v2 receives copy of production traffic
3. v1 predictions used for responses (no user impact)
4. Compare v1 and v2 predictions offline
5. Analyze differences, measure v2 accuracy
6. Promote v2 to production if performance acceptable

Use Cases:

• High-risk models (financial, healthcare, safety-critical)
• Need extensive testing before cutover
• Compare model behavior on real production data

For deployment architecture, see [references/deployment-strategies.md](references/deployment-strategies.md#shadow).

Tool Recommendations

Production-Ready Tools (High Adoption)

MLflow - Experiment Tracking & Model Registry

• GitHub Stars: 20,000+
• Trust Score: 95/100
• Use Cases: Experiment tracking, model registry, model serving
• Strengths: Open-source, framework-agnostic, self-hosted option
• Getting Started: pip install mlflow && mlflow server

Feast - Feature Store

• GitHub Stars: 5,000+
• Trust Score: 85/100
• Use Cases: Online/offline feature serving, point-in-time correctness
• Strengths: Cloud-agnostic, most popular open-source feature store
• Getting Started: pip install feast && feast init

Seldon Core - Model Serving (Advanced)

• GitHub Stars: 4,000+
• Trust Score: 85/100
• Use Cases: Kubernetes-native serving, advanced deployment patterns
• Strengths: Canary, A/B testing, MAB, explainability
• Limitation: High complexity, steep learning curve

KServe - Model Serving (CNCF Standard)

• GitHub Stars: 3,500+
• Trust Score: 85/100
• Use Cases: Standardized serving API, serverless scaling
• Strengths: CNCF project, Knative integration, growing adoption
• Limitation: Kubernetes required

BentoML - Model Serving (Simplicity)

• GitHub Stars: 6,000+
• Trust Score: 80/100
• Use Cases: Easy packaging, Python-first deployment
• Strengths: Lowest learning curve, excellent developer experience
• Limitation: Fewer advanced features than Seldon/KServe

Kubeflow Pipelines - ML Orchestration

• GitHub Stars: 14,000+ (Kubeflow project)
• Trust Score: 90/100
• Use Cases: ML-specific pipelines, Kubernetes-native workflows
• Strengths: ML-native, component reusability, Katib integration
• Limitation: Kubernetes required, steep learning curve

Weights & Biases - Experiment Tracking (SaaS)

• Trust Score: 90/100
• Use Cases: Team collaboration, advanced visualization, hyperparameter tuning
• Strengths: Best-in-class UI, integrated Sweeps, strong community
• Limitation: SaaS pricing, no self-hosted free tier

For detailed tool comparisons, see [references/tool-recommendations.md](references/tool-recommendations.md).

Tool Stack Recommendations by Organization

Startup (Cost-Optimized, Simple):

• Experiment Tracking: MLflow (free, self-hosted)
• Feature Store: None initially → Feast when needed
• Model Serving: BentoML (easy) or cloud functions
• Orchestration: Prefect or cron jobs
• Monitoring: Basic logging + Prometheus

Growth Company (Balanced):

• Experiment Tracking:

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ancoleman
• Source: ancoleman/ai-design-components
• License: MIT

Install and usage instructions live in the source repository linked above.