senior-data-engineer

# Senior Data Engineer World-class senior data engineer skill for production-grade AI/ML/Data systems. ## Quick Start ### Main Capabilities ```bash # Core Tool 1 python scripts/pipeline_orchestrator.py --input data/ --output results/ # Core Tool 2 python scripts/data_quality_validator.py --target project/ --analyze # Core Tool 3 python scripts/etl_performance_optimizer.py --config config.yaml --deploy ``` ## Core Expertise This skill covers world-class capabilities in: - Advanced production patterns and architectures - Scalable system design and implementation - Performance optimization at scale - MLOps and DataOps best practices - Real-time processing and inference - Distributed computing frameworks - Model deployment and monitoring - Security and compliance - Cost optimization - Team leadership and mentoring ## Tech Stack **Languages:** Python, SQL, R, Scala, Go **ML Frameworks:** PyTorch, TensorFlow, Scikit-learn, XGBoost **Data Tools:** Spark, Airflow, dbt, Kafka, Databricks **LLM Frameworks:** LangChain, LlamaIndex, DSPy **Deployment:** Docker, Kubernetes, AWS/GCP/Azure **Monitoring:** MLflow, Weights & Biases, Prometheus **Databases:** PostgreSQL, BigQuery, Snowflake, Pinecone ## Reference Documentation ### 1. Data Pipeline Architecture Comprehensive guide available in `references/data_pipeline_architecture.md` covering: - Advanced patterns and best practices - Production implementation strategies - Performance optimization techniques - Scalability cons...