[k8s] kubernetes Upgrade-master

Control plane upgrade (hk8s)

• Kubernetes cluster upgrade
  • kubeadm, kubelet, kubectl을 각각 업그레이드
• Control-plane Upgrade
  1. Upgrade할 master에 접속
  2. 업그레이드 할 버전 확인
  3. kubeadm 업그레이드
  4. 노드 드레인 : console이나 master에서 실행
  5. kubelet과 kubectl 업그레이드
  6. 노드 uncordon

 

Solution

• Control-plane Upgrade (https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#determine-which-version-to-upgrade-to)

1. Upgrade할 master에 접속

# hk8s-master 노드에 접속
$ kubectl config use-context hk8s 
Switched to context "hk8s".
$ kubectl config current-context 
hk8s
$ kubectl get nodes
NAME           STATUS     ROLES           AGE    VERSION
hk8s-master    Ready      control-plane   130d   v1.26.0
hk8s-worker1   Ready      <none>          130d   v1.26.0
hk8s-worker2   NotReady   <none>          130d   v1.26.0
$ ssh hk8s-master 
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-60-generic x86_64)

# os 확인
$ sudo cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
...

2. 업그레이드 할 버전 확인

# 업그레이드할 버전 확인하고 결정
$ apt update \
  apt-cache madison kubeadm
...
79 packages can be upgraded. Run 'apt list --upgradable' to see them.
   kubeadm |  1.27.3-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.27.2-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubeadm |  1.27.1-00 | https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
...

3. kubeadm 업그레이드

# kubeadm 1.27.1-00 다운로드
$ apt-mark unhold kubeadm && \
  apt-get update && apt-get install -y kubeadm=1.27.1-00 && \
  apt-mark hold kubeadm
...
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 78 not upgraded.
Need to get 9928 kB of archives.
...

# 다운로드 하려는 버전이 잘 받아졌는지 확인
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", 
GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", 
BuildDate:"2023-04-14T13:20:04Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}

# 업그레이드 계획 확인
$ kubeadm upgrade plan v1.27.1
Components that must be upgraded manually after you have upgraded the control plane with 
'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     3 x v1.26.0   1.27.1

Upgrade to the latest version in the v1.26 series:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.26.2   1.27.1
kube-controller-manager   v1.26.2   1.27.1
kube-scheduler            v1.26.2   1.27.1
kube-proxy                v1.26.2   1.27.1
CoreDNS                   v1.9.3    v1.10.1
etcd                      3.5.6-0   3.5.7-0

You can now apply the upgrade by executing the following command:

	kubeadm upgrade apply 1.27.1
_____________________________________________________________________

# 업그레이드 할 버전을 선택하여 명령을 실행
$ sudo kubeadm upgrade apply v1.27.1
...
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
...
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.27.1". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets
if you haven't already done so.

4. 노드 드레인 : console이나 master에서 실행

# unschedulable로 표시하고 워크로드를 제거하여 유지 관리를 위해 노드를 준비
$ kubectl drain hk8s-master --ignore-daemonsets
node/hk8s-master cordoned
Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-99wtw, calico-system/ ...
node/hk8s-master drained

※ drain 하던 과정에서 'exevicting pod calico-system/calico-typha-5d975894b7-7g9zm
error when evicting pods/"calico-typha-5d975894b7-7g9zm" -n "calico-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.' 라는 에러가 발생하였다.

• hk8s-master의 component.yaml 파일의 maxUnavailable 값을 0에서 1로 변경해주었더니 에러가 발생하지 않았다.
• https://github.com/projectcalico/calico/issues/4211

# ~/component.yaml
...
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 1
...

5. kubelet과 kubectl 업그레이드

# 컨트롤플레인에서  kublet과 kubectl을 업그레이드 한다.
$ apt-mark unhold kubelet kubectl && \
  apt-get update && apt-get install -y kubelet=1.27.1-00 kubectl=1.27.1-00 && \
  apt-mark hold kubelet kubectl
...
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 77 not upgraded.
...

# kubelet 다시 시작
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet

6. 노드 uncordon

# hk8s-master node의 Status와 version확인
$ kubectl get nodes
NAME           STATUS                     ROLES           AGE    VERSION
hk8s-master    Ready,SchedulingDisabled   control-plane   130d   v1.27.1
hk8s-worker1   Ready                      <none>          130d   v1.26.0
hk8s-worker2   NotReady                   <none>          130d   v1.26.0

# 노드를 스케줄 가능하도록 표시하여 다시 온라인 상태로 전환
$ kubectl uncordon hk8s-master
node/hk8s-master uncordoned
$ kubectl get nodes
NAME           STATUS     ROLES           AGE    VERSION
hk8s-master    Ready      control-plane   130d   v1.27.1
hk8s-worker1   Ready      <none>          130d   v1.26.0
hk8s-worker2   NotReady   <none>          130d   v1.26.0