cluster-operations

# Skill: Cluster Operations > **Expertise:** Safe day-2 operations on self-hosted Kubernetes clusters — node drain, etcd ops, cert rotation. ## When to load When draining nodes for maintenance, rotating certificates, backing up etcd, or troubleshooting control plane issues. ## Node Lifecycle Operations ```bash # --- CORDON (stop scheduling new pods, don't evict existing) --- kubectl cordon <node-name> # Use case: pre-drain notification, temporary maintenance hold # --- DRAIN (evict all pods, mark unschedulable) --- kubectl drain <node-name> \ --ignore-daemonsets \ # DaemonSet pods can't be evicted --delete-emptydir-data \ # required for pods using emptyDir --grace-period=60 \ # give pods time to shut down cleanly --timeout=300s # abort if takes > 5 minutes # After drain: node is unschedulable and empty (except daemonsets) # --- UNCORDON (return to service) --- kubectl uncordon <node-name> # --- Verify node is empty before maintenance --- kubectl get pods -A --field-selector=spec.nodeName=<node-name> ``` ## etcd Backup (bare-metal / kubeadm) ```bash # --- Take snapshot (run on a control plane node) --- ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \ --key=/etc/kubernetes/pki/etcd/healthcheck-client.key # --- Verify snapshot --- ETCDCTL_API=3 etcd...