CRI-O기반의 k8s설치

사전사항

1. OS환경설정
    

    

    

    

   1

   $> swapoff -a

   2

   ​

   3

   $> cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

   4

   br_netfilter

   5

   EOF

   6

   ​

   7

   $> cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf

   8

   net.bridge.bridge-nf-call-ip6tables = 1

   9

   net.bridge.bridge-nf-call-iptables = 1

   10

   EOF

   11

   ​

   12

   $> sudo sysctl --system

    

    

    
2. crio / kubernetees 패키지 리포지터리 구성
    

    

    

    

    

   1

   $> cat /etc/yum.repos.d/libcontainers.repo

   2

   [devel_kubic_libcontainers_stable]

   3

   name=Stable Releases of Upstream github.com/containers packages (CentOS_8)

   4

   type=rpm-md

   5

   baseurl=https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/

   6

   gpgcheck=1

   7

   gpgkey=https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/repodata/repomd.xml.key

   8

   enabled=1

    

    

    
   
   
   
    

    

    

    

    

   1

   $> cat /etc/yum.repos.d/cri-o-1.23.repo

   2

   [devel_kubic_libcontainers_stable_cri-o_1.23]

   3

   name=devel:kubic:libcontainers:stable:cri-o:1.23 (CentOS_8)

   4

   type=rpm-md

   5

   baseurl=https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.23/CentOS_8/

   6

   gpgcheck=1

   7

   gpgkey=https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.23/CentOS_8/repodata/repomd.xml.key

   8

   enabled=1

    

    

    
    
    

    

    

    

    

   1

   $> cat /etc/yum.repos.d/kubernetes.repo

   2

   [kubernetes]

   3

   name=Kubernetes

   4

   baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch

   5

   enabled=1

   6

   gpgcheck=1

   7

   repo_gpgcheck=1

   8

   gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

    

    

    
    
3. 패키지 설치
    

    

    

    

   1

   $> yum install kubelet kubeadm kubectl libcgroup cri-o cri-tools -y

   2

   $> systemctl enable crio --now

   3

   $> systemctl enable kubelet

    

    

    

클러스터 생성 (control palin 1번에서만 수행)

1. kubeadm 클러스터 생성
    

    

    

    

    

   1

   $> kubeadm init --control-plane-endpoint 172.21.107.238:6443 --pod-network-cidr 10.250.0.0/16 --ignore-preflight-errors=all --upload-certs

   2

   ​

   3

   ## 결과값중에 control / worker 노드별 join 명령이 다르기 때문에 별도로 복사해놓어야 함

    

    

    
   
   
   ##Control node용
    

    

    

    

   1

   $> kubeadm join 172.21.107.238:6443 --token abcd \

   2

          --discovery-token-ca-cert-hash sha256:yyy \

   3

          --control-plane --certificate-key zzz

    

    

    
   
   ## Worker Node용
    

    

    

    

   1

   $> kubeadm join 172.21.107.238:6443 --token abcd \

   2

          --discovery-token-ca-cert-hash sha256:yyyy \

   3

    --ignore-preflight-errors=all

    

    

    
    
2. 인증서 정보 복사
    

    

    

    

   1

   $> mkdir -p $HOME/.kube

   2

   $> /bin/cp /etc/kubernetes/admin.conf $HOME/.kube/config

   3

   $> chown $(id -u):$(id -g) $HOME/.kube/config

   4

   $> export KUBECONFIG=/etc/kubernetes/admin.conf

    

    

    
3. CNI 설치(Calico)
    

    

    

    

   1

   $> curl https://projectcalico.docs.tigera.io/manifests/calico.yaml -O

   2

   $> kubectl apply -f calico.yaml

    

    

    

클러스터 연동

1. 타 Control plain 연동 (Control Plain 한대씩 순차 작업 수행)
    

    

    

    

   1

   $> kubeadm join 172.21.107.238:6443 --token abcd \

   2

          --discovery-token-ca-cert-hash sha256:yyy \

   3

          --control-plane --certificate-key zzz

    

    

    
2. 노드 연동 확인 (Control plain에서 수행)
    

    

    

    

   1

   $> kubectl get no

   2

   NAME                         STATUS   ROLES                  AGE    VERSION

   3

   k8stesttx-k8s-master-dev01   Ready    control-plane,master   3h6m   v1.23.5

   4

   k8stesttx-k8s-master-dev02   Ready    control-plane,master   3h6m   v1.23.5

   5

   k8stesttx-k8s-master-dev03   Ready    control-plane,master   3h6m   v1.23.5

    

    

    
3. Worker Node  연동
    

    

    

    

   1

   $> kubeadm join 172.21.107.238:6443 --token abcd \

   2

          --discovery-token-ca-cert-hash sha256:yyyy \

   3

    --ignore-preflight-errors=all

    

    

    
4. 노드 연동 확인 (Control plain에서 수행)
    

    

    

    

   1

   $> kubectl get no

   2

   NAME                         STATUS   ROLES                  AGE    VERSION

   3

   ...

   4

   k8stesttx-k8s-worker-dev01   Ready    <none>                 40m    v1.23.5

   5

   k8stesttx-k8s-worker-dev02   Ready    <none>                 40m    v1.23.5

   6

   k8stesttx-k8s-worker-dev03   Ready    <none>                 40m    v1.23.5

    

    

    

k8s 인증서 10년으로 연장

 

1. 인증서 연장 스크립트
    

    

    

    

    

   1

   #!/usr/bin/env bash

   2

   ​

   3

   set -o errexit

   4

   set -o pipefail

   5

   # set -o xtrace

   6

   ​

   7

   # set output color

   8

   NC='\033[0m'

   9

   RED='\033[31m'

   10

   GREEN='\033[32m'

   11

   YELLOW='\033[33m'

   12

   BLUE='\033[34m'

   13

   ​

   14

   log::err() {

   15

    printf "[$(date +'%Y-%m-%dT%H:%M:%S.%2N%z')][${RED}ERROR${NC}] %b\n" "$@"

   16

   }

   17

   ​

   18

   log::info() {

   19

    printf "[$(date +'%Y-%m-%dT%H:%M:%S.%2N%z')][INFO] %b\n" "$@"

   20

   }

   21

   ​

   22

   log::warning() {

   23

    printf "[$(date +'%Y-%m-%dT%H:%M:%S.%2N%z')][${YELLOW}WARNING${NC}] \033[0m%b\n" "$@"

   24

   }

   25

   ​

   26

   check_file() {

   27

    if [[ ! -r ${1} ]]; then

   28

       log::err "can not find ${1}"

   29

      exit 1

   30

    fi

   31

   }

   32

   ​

   33

   # get x509v3 subject alternative name from the old certificate

   34

   cert::get_subject_alt_name() {

   35

    local cert=${1}.crt

   36

    local alt_name

   37

   ​

   38

     check_file "${cert}"

   39

    alt_name=$(openssl x509 -text -noout -in "${cert}" | grep -A1 'Alternative' | tail -n1 | sed 's/[[:space:]]*Address//g')

   40

    printf "%s\n" "${alt_name}"

   41

   }

   42

   ​

   43

   # get subject from the old certificate

   44

   cert::get_subj() {

   45

    local cert=${1}.crt

   46

    local subj

   47

   ​

   48

     check_file "${cert}"

   49

    subj=$(openssl x509 -text -noout -in "${cert}" | grep "Subject:" | sed 's/Subject:/\//g;s/\,/\//;s/[[:space:]]//g')

   50

    printf "%s\n" "${subj}"

   51

   }

   52

   ​

   53

   cert::backup_file() {

   54

    local file=${1}

   55

    if [[ ! -e ${file}.old-$(date +%Y%m%d) ]]; then

   56

       cp -rp "${file}" "${file}.old-$(date +%Y%m%d)"

   57

       log::info "backup ${file} to ${file}.old-$(date +%Y%m%d)"

   58

    else

   59

       log::warning "does not backup, ${file}.old-$(date +%Y%m%d) already exists"

   60

    fi

   61

   }

   62

   ​

   63

   # check certificate expiration

   64

   cert::check_cert_expiration() {

   65

    local cert=${1}.crt

   66

    local cert_expires

   67

   ​

   68

    cert_expires=$(openssl x509 -text -noout -in "${cert}" | awk -F ": " '/Not After/{print$2}')

   69

    printf "%s\n" "${cert_expires}"

   70

   }

   71

   ​

   72

   # check kubeconfig expiration

   73

   cert::check_kubeconfig_expiration() {

   74

    local config=${1}.conf

   75

    local cert

   76

    local cert_expires

   77

   ​

   78

    cert=$(grep "client-certificate-data" "${config}" | awk '{print$2}' | base64 -d)

   79

    cert_expires=$(openssl x509 -text -noout -in <(printf "%s" "${cert}") | awk -F ": " '/Not After/{print$2}')

   80

    printf "%s\n" "${cert_expires}"

   81

   }

   82

   ​

   83

   # check etcd certificates expiration

   84

   cert::check_etcd_certs_expiration() {

   85

    local cert

   86

    local certs

   87

   ​

   88

    certs=(

   89

      "${ETCD_CERT_CA}"

   90

      "${ETCD_CERT_SERVER}"

   91

      "${ETCD_CERT_PEER}"

   92

      "${ETCD_CERT_HEALTHCHECK_CLIENT}"

   93

      "${ETCD_CERT_APISERVER_ETCD_CLIENT}"

   94

    )

   95

   ​

   96

    for cert in "${certs[@]}"; do

   97

      if [[ ! -r ${cert} ]]; then

   98

        printf "%-50s%-30s\n" "${cert}.crt" "$(cert::check_cert_expiration "${cert}")"

   99

      fi

   100

    done

   101

   }

   102

   ​

   103

   # check master certificates expiration

   104

   cert::check_master_certs_expiration() {

   105

    local certs

   106

    local kubeconfs

   107

    local cert

   108

    local conf

   109

   ​

   110

    certs=(

   111

      "${CERT_CA}"

   112

      "${CERT_APISERVER}"

   113

      "${CERT_APISERVER_KUBELET_CLIENT}"

   114

      "${FRONT_PROXY_CA}"

   115

      "${FRONT_PROXY_CLIENT}"

   116

    )

   117

   ​

   118

    kubeconfs=(

   119

      "${CONF_CONTROLLER_MANAGER}"

   120

      "${CONF_SCHEDULER}"

   121

      "${CONF_ADMIN}"

   122

    )

   123

   ​

   124

    printf "%-50s%-30s\n" "CERTIFICATE" "EXPIRES"

   125

   ​

   126

    for conf in "${kubeconfs[@]}"; do

   127

      if [[ ! -r ${conf} ]]; then

   128

        printf "%-50s%-30s\n" "${conf}.config" "$(cert::check_kubeconfig_expiration "${conf}")"

   129

      fi

   130

    done

   131

   ​

   132

    for cert in "${certs[@]}"; do

   133

      if [[ ! -r ${cert} ]]; then

   134

        printf "%-50s%-30s\n" "${cert}.crt" "$(cert::check_cert_expiration "${cert}")"

   135

      fi

   136

    done

   137

   }

   138

   ​

   139

   # check all certificates expiration

   140

   cert::check_all_expiration() {

   141

     cert::check_master_certs_expiration

   142

     cert::check_etcd_certs_expiration

   143

   }

   144

   ​

   145

   # generate certificate whit client, server or peer

   146

   # Args:

   147

   #   $1 (the name of certificate)

   148

   #   $2 (the type of certificate, must be one of client, server, peer)

   149

   #   $3 (the subject of certificates)

   150

   #   $4 (the validity of certificates) (days)

   151

   #   $5 (the name of ca)

   152

   #   $6 (the x509v3 subject alternative name of certificate when the type of certificate is server or peer)

   153

   cert::gen_cert() {

   154

    local cert_name=${1}

   155

    local cert_type=${2}

   156

    local subj=${3}

   157

    local cert_days=${4}

   158

    local ca_name=${5}

   159

    local alt_name=${6}

   160

    local ca_cert=${ca_name}.crt

   161

    local ca_key=${ca_name}.key

   162

    local cert=${cert_name}.crt

   163

    local key=${cert_name}.key

   164

    local csr=${cert_name}.csr

   165

    local common_csr_conf='distinguished_name = dn\n[dn]\n[v3_ext]\nkeyUsage = critical, digitalSignature, keyEncipherment\n'

   166

   ​

   167

    for file in "${ca_cert}" "${ca_key}" "${cert}" "${key}"; do

   168

       check_file "${file}"

   169

    done

   170

   ​

   171

    case "${cert_type}" in

   172

     client)

   173

      csr_conf=$(printf "%bextendedKeyUsage = clientAuth\n" "${common_csr_conf}")

   174

       ;;

   175

     server)

   176

      csr_conf=$(printf "%bextendedKeyUsage = serverAuth\nsubjectAltName = %b\n" "${common_csr_conf}" "${alt_name}")

   177

       ;;

   178

     peer)

   179

      csr_conf=$(printf "%bextendedKeyUsage = serverAuth, clientAuth\nsubjectAltName = %b\n" "${common_csr_conf}" "${alt_name}")

   180

       ;;

   181

     *)

   182

       log::err "unknow, unsupported certs type: ${YELLOW}${cert_type}${NC}, supported type: client, server, peer"

   183

      exit 1

   184

       ;;

   185

    esac

   186

   ​

   187

    # gen csr

   188

    openssl req -new -key "${key}" -subj "${subj}" -reqexts v3_ext \

   189

      -config <(printf "%b" "${csr_conf}") \

   190

      -out "${csr}" >/dev/null 2>&1

   191

    # gen cert

   192

    openssl x509 -in "${csr}" -req -CA "${ca_cert}" -CAkey "${ca_key}" -CAcreateserial -extensions v3_ext \

   193

      -extfile <(printf "%b" "${csr_conf}") \

   194

      -days "${cert_days}" -out "${cert}" >/dev/null 2>&1

   195

   ​

   196

     rm -f "${csr}"

   197

   }

   198

   ​

   199

   cert::update_kubeconf() {

   200

    local cert_name=${1}

   201

    local kubeconf_file=${cert_name}.conf

   202

    local cert=${cert_name}.crt

   203

    local key=${cert_name}.key

   204

    local subj

   205

    local cert_base64

   206

   ​

   207

     check_file "${kubeconf_file}"

   208

    # get the key from the old kubeconf

   209

    grep "client-key-data" "${kubeconf_file}" | awk '{print$2}' | base64 -d >"${key}"

   210

    # get the old certificate from the old kubeconf

   211

    grep "client-certificate-data" "${kubeconf_file}" | awk '{print$2}' | base64 -d >"${cert}"

   212

    # get subject from the old certificate

   213

    subj=$(cert::get_subj "${cert_name}")

   214

     cert::gen_cert "${cert_name}" "client" "${subj}" "${CERT_DAYS}" "${CERT_CA}"

   215

    # get certificate base64 code

   216

    cert_base64=$(base64 -w 0 "${cert}")

   217

   ​

   218

    # set certificate base64 code to kubeconf

   219

    sed -i 's/client-certificate-data:.*/client-certificate-data: '"${cert_base64}"'/g' "${kubeconf_file}"

   220

   ​

   221

     rm -f "${cert}"

   222

     rm -f "${key}"

   223

   }

   224

   ​

   225

   cert::update_etcd_cert() {

   226

    local subj

   227

    local subject_alt_name

   228

    local cert

   229

   ​

   230

    # generate etcd server,peer certificate

   231

    # /etc/kubernetes/pki/etcd/server

   232

    # /etc/kubernetes/pki/etcd/peer

   233

    for cert in ${ETCD_CERT_SERVER} ${ETCD_CERT_PEER}; do

   234

      subj=$(cert::get_subj "${cert}")

   235

      subject_alt_name=$(cert::get_subject_alt_name "${cert}")

   236

       cert::gen_cert "${cert}" "peer" "${subj}" "${CERT_DAYS}" "${ETCD_CERT_CA}" "${subject_alt_name}"

   237

       log::info "${GREEN}updated ${BLUE}${cert}.conf${NC}"

   238

    done

   239

   ​

   240

    # generate etcd healthcheck-client,apiserver-etcd-client certificate

   241

    # /etc/kubernetes/pki/etcd/healthcheck-client

   242

    # /etc/kubernetes/pki/apiserver-etcd-client

   243

    for cert in ${ETCD_CERT_HEALTHCHECK_CLIENT} ${ETCD_CERT_APISERVER_ETCD_CLIENT}; do

   244

      subj=$(cert::get_subj "${cert}")

   245

       cert::gen_cert "${cert}" "client" "${subj}" "${CERT_DAYS}" "${ETCD_CERT_CA}"

   246

       log::info "${GREEN}updated ${BLUE}${cert}.conf${NC}"

   247

    done

   248

   ​

   249

    # restart etcd

   250

    docker ps | awk '/k8s_etcd/{print$1}' | xargs -r -I '{}' docker restart {} >/dev/null 2>&1 || true

   251

     log::info "restarted etcd"

   252

   }

   253

   ​

   254

   cert::update_master_cert() {

   255

    local subj

   256

    local subject_alt_name

   257

    local conf

   258

   ​

   259

    # generate apiserver server certificate

   260

    # /etc/kubernetes/pki/apiserver

   261

    subj=$(cert::get_subj "${CERT_APISERVER}")

   262

    subject_alt_name=$(cert::get_subject_alt_name "${CERT_APISERVER}")

   263

     cert::gen_cert "${CERT_APISERVER}" "server" "${subj}" "${CERT_DAYS}" "${CERT_CA}" "${subject_alt_name}"

   264

     log::info "${GREEN}updated ${BLUE}${CERT_APISERVER}.crt${NC}"

   265

   ​

   266

    # generate apiserver-kubelet-client certificate

   267

    # /etc/kubernetes/pki/apiserver-kubelet-client

   268

    subj=$(cert::get_subj "${CERT_APISERVER_KUBELET_CLIENT}")

   269

     cert::gen_cert "${CERT_APISERVER_KUBELET_CLIENT}" "client" "${subj}" "${CERT_DAYS}" "${CERT_CA}"

   270

     log::info "${GREEN}updated ${BLUE}${CERT_APISERVER_KUBELET_CLIENT}.crt${NC}"

   271

   ​

   272

    # generate kubeconf for controller-manager,scheduler and kubelet

   273

    # /etc/kubernetes/controller-manager,scheduler,admin,kubelet.conf

   274

    for conf in ${CONF_CONTROLLER_MANAGER} ${CONF_SCHEDULER} ${CONF_ADMIN} ${CONF_KUBELET}; do

   275

      if [[ ${conf##*/} == "kubelet" ]]; then

   276

        # https://github.com/kubernetes/kubeadm/issues/1753

   277

        set +e

   278

         grep kubelet-client-current.pem /etc/kubernetes/kubelet.conf >/dev/null 2>&1

   279

        kubelet_cert_auto_update=$?

   280

        set -e

   281

        if [[ "$kubelet_cert_auto_update" == "0" ]]; then

   282

           log::info "does not need to update kubelet.conf"

   283

          continue

   284

        fi

   285

      fi

   286

   ​

   287

      # update kubeconf

   288

      cert::update_kubeconf "${conf}"

   289

       log::info "${GREEN}updated ${BLUE}${conf}.conf${NC}"

   290

   ​

   291

      # copy admin.conf to ${HOME}/.kube/config

   292

      if [[ ${conf##*/} == "admin" ]]; then

   293

         mkdir -p "${HOME}/.kube"

   294

        local config=${HOME}/.kube/config

   295

        local config_backup

   296

        config_backup=${HOME}/.kube/config.old-$(date +%Y%m%d)

   297

        if [[ -f ${config} ]] && [[ ! -f ${config_backup} ]]; then

   298

           cp -fp "${config}" "${config_backup}"

   299

           log::info "backup ${config} to ${config_backup}"

   300

        fi

   301

         cp -fp "${conf}.conf" "${HOME}/.kube/config"

   302

         log::info "copy the admin.conf to ${HOME}/.kube/config"

   303

      fi

   304

    done

   305

   ​

   306

    # generate front-proxy-client certificate

   307

    # /etc/kubernetes/pki/front-proxy-client

   308

    subj=$(cert::get_subj "${FRONT_PROXY_CLIENT}")

   309

     cert::gen_cert "${FRONT_PROXY_CLIENT}" "client" "${subj}" "${CERT_DAYS}" "${FRONT_PROXY_CA}"

   310

     log::info "${GREEN}updated ${BLUE}${FRONT_PROXY_CLIENT}.crt${NC}"

   311

   ​

   312

    # restart apiserver, controller-manager, scheduler and kubelet

   313

    for item in "apiserver" "controller-manager" "scheduler"; do

   314

       docker ps | awk '/k8s_kube-'${item}'/{print$1}' | xargs -r -I '{}' docker restart {} >/dev/null 2>&1 || true

   315

       log::info "restarted ${item}"

   316

    done

   317

     systemctl restart kubelet || true

   318

     log::info "restarted kubelet"

   319

   }

   320

   ​

   321

   main() {

   322

    local node_type=$1

   323

   ​

   324

    CERT_DAYS=3650

   325

   ​

   326

    KUBE_PATH=/etc/kubernetes

   327

    PKI_PATH=${KUBE_PATH}/pki

   328

   ​

   329

    # master certificates path

   330

    # apiserver

   331

    CERT_CA=${PKI_PATH}/ca

   332

    CERT_APISERVER=${PKI_PATH}/apiserver

   333

    CERT_APISERVER_KUBELET_CLIENT=${PKI_PATH}/apiserver-kubelet-client

   334

    CONF_CONTROLLER_MANAGER=${KUBE_PATH}/controller-manager

   335

    CONF_SCHEDULER=${KUBE_PATH}/scheduler

   336

    CONF_ADMIN=${KUBE_PATH}/admin

   337

    CONF_KUBELET=${KUBE_PATH}/kubelet

   338

    # front-proxy

   339

    FRONT_PROXY_CA=${PKI_PATH}/front-proxy-ca

   340

    FRONT_PROXY_CLIENT=${PKI_PATH}/front-proxy-client

   341

   ​

   342

    # etcd certificates path

   343

    ETCD_CERT_CA=${PKI_PATH}/etcd/ca

   344

    ETCD_CERT_SERVER=${PKI_PATH}/etcd/server

   345

    ETCD_CERT_PEER=${PKI_PATH}/etcd/peer

   346

    ETCD_CERT_HEALTHCHECK_CLIENT=${PKI_PATH}/etcd/healthcheck-client

   347

    ETCD_CERT_APISERVER_ETCD_CLIENT=${PKI_PATH}/apiserver-etcd-client

   348

   ​

   349

    case ${node_type} in

   350

    # etcd)

   351

    # # update etcd certificates

   352

    #   cert::update_etcd_cert

   353

    # ;;

   354

    master)

   355

      # check certificates expiration

   356

      cert::check_master_certs_expiration

   357

      # backup $KUBE_PATH to $KUBE_PATH.old-$(date +%Y%m%d)

   358

      cert::backup_file "${KUBE_PATH}"

   359

      # update master certificates and kubeconf

   360

      log::info "${GREEN}updating...${NC}"

   361

       cert::update_master_cert

   362

       log::info "${GREEN}done!!!${NC}"

   363

      # check certificates expiration after certificates updated

   364

      cert::check_master_certs_expiration

   365

       ;;

   366

     all)

   367

      # check certificates expiration

   368

      cert::check_all_expiration

   369

      # backup $KUBE_PATH to $KUBE_PATH.old-$(date +%Y%m%d)

   370

      cert::backup_file "${KUBE_PATH}"

   371

      # update etcd certificates

   372

      log::info "${GREEN}updating...${NC}"

   373

       cert::update_etcd_cert

   374

      # update master certificates and kubeconf

   375

      cert::update_master_cert

   376

       log::info "${GREEN}done!!!${NC}"

   377

      # check certificates expiration after certificates updated

   378

      cert::check_all_expiration

   379

       ;;

   380

     check)

   381

      # check certificates expiration

   382

      cert::check_all_expiration

   383

       ;;

   384

     *)

   385

       log::err "unknown, unsupported cert type: ${node_type}, supported type: \"all\", \"master\""

   386

      printf "Documentation: https://github.com/yuyicai/update-kube-cert

   387

     example:

   388

       '\033[32m./update-kubeadm-cert.sh all\033[0m' update all etcd certificates, master certificates and kubeconf

   389

         /etc/kubernetes

   390

         ├── admin.conf

   391

         ├── controller-manager.conf

   392

         ├── scheduler.conf

   393

         ├── kubelet.conf

   394

         └── pki

   395

             ├── apiserver.crt

   396

             ├── apiserver-etcd-client.crt

   397

             ├── apiserver-kubelet-client.crt

   398

             ├── front-proxy-client.crt

   399

             └── etcd

   400

                 ├── healthcheck-client.crt

   401

                 ├── peer.crt

   402

                 └── server.crt

   403

   ​

   404

       '\033[32m./update-kubeadm-cert.sh master\033[0m' update only master certificates and kubeconf

   405

         /etc/kubernetes

   406

         ├── admin.conf

   407

         ├── controller-manager.conf

   408

         ├── scheduler.conf

   409

         ├── kubelet.conf

   410

         └── pki

   411

             ├── apiserver.crt

   412

             ├── apiserver-kubelet-client.crt

   413

             └── front-proxy-client.crt

   414

   "

   415

      exit 1

   416

       ;;

   417

    esac

   418

   }

   419

   ​

   420

   main "$@"

    

    

    
2. 업데이트 전 인증서 정보 확인
    

    

    

    

   1

   $> kubeadm certs check-expiration

   2

   [check-expiration] Reading configuration from the cluster...

   3

   [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

   4

   ​

   5

   CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED

   6

   admin.conf                 Apr 17, 2023 06:09 UTC   364d            ca                      no

   7

   apiserver                  Apr 17, 2023 06:09 UTC   364d            ca                      no

   8

   apiserver-etcd-client      Apr 17, 2023 06:09 UTC   364d            etcd-ca                 no

   9

   apiserver-kubelet-client   Apr 17, 2023 06:09 UTC   364d            ca                      no

   10

   controller-manager.conf    Apr 17, 2023 06:09 UTC   364d            ca                      no

   11

   etcd-healthcheck-client    Apr 17, 2023 06:09 UTC   364d            etcd-ca                 no

   12

   etcd-peer                  Apr 17, 2023 06:09 UTC   364d            etcd-ca                 no

   13

   etcd-server                Apr 17, 2023 06:09 UTC   364d            etcd-ca                 no

   14

   front-proxy-client         Apr 17, 2023 06:09 UTC   364d            front-proxy-ca          no

   15

   scheduler.conf             Apr 17, 2023 06:09 UTC   364d            ca                      no

   16

   ​

   17

   CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED

   18

   ca                      Apr 17, 2032 04:17 UTC   9y              no

   19

   etcd-ca                 Apr 17, 2032 04:17 UTC   9y              no

   20

   front-proxy-ca          Apr 17, 2032 04:17 UTC   9y              no

    

    

    
3. 인증서 업데이트 (Control Plain 1대씩 순차 작업 수행, 서버단위로 30초 가량 대기 필요)
    

    

    

    

   1

   $> chmod +x cert_update.sh

   2

   $> ./cert_update.sh

   3

   ...

    

    

    
4. 인증서 갱신정보 확인
    

    

    

    

   1

   $> kubeadm certs check-expiration

   2

   [check-expiration] Reading configuration from the cluster...

   3

   [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

   4

   ​

   5

   CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED

   6

   admin.conf                 Apr 17, 2032 06:09 UTC   9y              ca                      no

   7

   apiserver                  Apr 17, 2032 06:09 UTC   9y              ca                      no

   8

   apiserver-etcd-client      Apr 17, 2032 06:09 UTC   9y              etcd-ca                 no

   9

   apiserver-kubelet-client   Apr 17, 2032 06:09 UTC   9y              ca                      no

   10

   controller-manager.conf    Apr 17, 2032 06:09 UTC   9y              ca                      no

   11

   etcd-healthcheck-client    Apr 17, 2032 06:09 UTC   9y              etcd-ca                 no

   12

   etcd-peer                  Apr 17, 2032 06:09 UTC   9y              etcd-ca                 no

   13

   etcd-server                Apr 17, 2032 06:09 UTC   9y              etcd-ca                 no

   14

   front-proxy-client         Apr 17, 2032 06:09 UTC   9y              front-proxy-ca          no

   15

   scheduler.conf             Apr 17, 2032 06:09 UTC   9y              ca                      no

   16

   ​

   17

   CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED

   18

   ca                      Apr 17, 2032 04:17 UTC   9y              no

   19

   etcd-ca                 Apr 17, 2032 04:17 UTC   9y              no

   20

   front-proxy-ca          Apr 17, 2032 04:17 UTC   9y              no

    

    

    

Reference

• https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
• https://github.com/yuyicai/update-kube-cert.git