ml-pipeline-automation

# ML Pipeline Automation Orchestrate end-to-end machine learning workflows from data ingestion to production deployment with production-tested Airflow, Kubeflow, and MLflow patterns. ## When to Use This Skill Load this skill when: - **Building ML Pipelines**: Orchestrating data → train → deploy workflows - **Scheduling Retraining**: Setting up automated model retraining schedules - **Experiment Tracking**: Tracking experiments, parameters, metrics across runs - **MLOps Implementation**: Building reproducible, monitored ML infrastructure - **Workflow Orchestration**: Managing complex multi-step ML workflows - **Model Registry**: Managing model versions and deployment lifecycle ## Quick Start: ML Pipeline in 5 Steps ```bash # 1. Install Airflow and MLflow (check for latest versions at time of use) pip install apache-airflow==3.1.5 mlflow==3.7.0 # Note: These versions are current as of December 2025 # Check PyPI for latest stable releases: https://pypi.org/project/apache-airflow/ # 2. Initialize Airflow database airflow db init # 3. Create DAG file: dags/ml_training_pipeline.py cat > dags/ml_training_pipeline.py << 'EOF' from airflow import DAG from airflow.operators.python import PythonOperator from datetime import datetime, timedelta default_args = { 'owner': 'ml-team', 'retries': 2, 'retry_delay': timedelta(minutes=5) } dag = DAG( 'ml_training_pipeline', default_args=default_args, schedule_interval='@daily', start_date=datetime(2025, 1, 1...