워커노드 제외방법

사전 설명

k8s환경에서 특정 노드 제외할때 그냥 삭제해도 kube scheduling에 의해 재배치가 있긴하지만 계획된 정지 작업같은 작업이 있는경우

보다 안정적으로 노드를 제외하려면 cordon / drain 절차를 거치면 좋습니다.

cordon / drain은 약간의 차이가 있는데, cordon의 경우 단순 스케쥴링을 제외하는 절차, drain의 경우 제외할 노드를 모두 제거하는 절차를 수행합니다. (drain은 cordon절차를 포함합니다.)

작업절차

1. cordon 설정

   1. 노드 정보 확인
      

      $>  kubectl get no
      NAME    STATUS   ROLES                  AGE   VERSION
      masr1   Ready    control-plane,master   37d   v1.23.7
      work1   Ready    <none>                 37d   v1.23.7
      work2   Ready    <none>                 37d   v1.23.7
      

   2. cordon 수행 (work1노드를 제외할껍니다.)
      

      $>  kubectl cordon work1
      node/work1 cordoned
      
      $>  kubectl get no
      NAME    STATUS                         ROLES                  AGE   VERSION
      masr1   Ready                          control-plane,master   37d   v1.23.7
      work1   Ready                          <none>                 37d   v1.23.7
      work2   Ready,SchedulingDisabled       <none>                 37d   v1.23.7
      

2. drain 수행

   1. 해당 노드에 daemonset으로 구동되는 pod가 있거나 local storage가 구동되고 있는 경우 drain 작업이 수행되지 않습니다.
      

      kubectl drain work2
      node/work2 already cordoned
      error: unable to drain node "work2", aborting command...
      
      There are pending nodes to be drained:
       k8sw2
      cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/calico-node-fb862, kube-system/kube-proxy-4qhsx, kube-system/nodelocaldns-6cptm
      cannot delete Pods with local storage (use --delete-emptydir-data to override): kubernetes-dashboard/dashboard-metrics-scraper-66dd8bdd86-6cnbv, kubernetes-dashboard/kubernetes-dashboard-844749bcff-mmpq2
      

   2. 강제로 drain 처리
      

      kubectl drain work2  --ignore-daemonsets --delete-emptydir-data
      node/work2 already cordoned
      WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-fb862, kube-system/kube-proxy-4qhsx, kube-system/nodelocaldns-6cptm
      evicting pod kubernetes-dashboard/kubernetes-dashboard-844749bcff-mmpq2
      evicting pod cert-manager/cert-manager-cainjector-75c94654d-r7trs
      evicting pod cert-manager/cert-manager-webhook-d4fd4f479-cr6z2
      evicting pod gitlab-agent/house-gitlab-agent-v1-586d6d6797-vm68t
      evicting pod kube-system/coredns-657959df74-j2zbq
      evicting pod kubernetes-dashboard/dashboard-metrics-scraper-66dd8bdd86-6cnbv
      pod/cert-manager-cainjector-75c94654d-r7trs evicted
      pod/cert-manager-webhook-d4fd4f479-cr6z2 evicted
      pod/house-gitlab-agent-v1-586d6d6797-vm68t evicted
      pod/kubernetes-dashboard-844749bcff-mmpq2 evicted
      pod/dashboard-metrics-scraper-66dd8bdd86-6cnbv evicted
      pod/coredns-657959df74-j2zbq evicted
      node/work2 evicted
      

3. 작업완료 

   1. 작업이 끝난노드를 다시 클러스터로 포함시키면 됩니다.
      

      kubectl uncordon work2 
      node/work2 already uncordoned
      

   2. 스케쥴링 활성화 확인
      

      $>  kubectl get no
      NAME    STATUS   ROLES                  AGE   VERSION
      masr1   Ready    control-plane,master   37d   v1.23.7
      work1   Ready    <none>                 37d   v1.23.7
      work2   Ready    <none>                 37d   v1.23.7
      

 

Reference

http://igoni.kr/books/linux/page/9be40

워커노드 제외방법 | igoni.kr