[k8s] ETCD Backup&Restore(1)

ETCD Backup&Restore

• ETCD를 호스팅 할 시스템에 ssh 로그인
• ETCD 툴의 설치 여부 확인
• ETCD 백업
• snapshot 파일을 이용해 복원했을 때 원래대로 복원되는지 확인하기 위해 현재 ETCD 상태를 수정
• ETCD 복원
• ETCD Pod에 복원된 etcd-data 위치를 적용하고 Pod 다시 시작

 

Solution

• ETCD를 호스팅 할 시스템에 ssh 로그인

$ kubectl config current-context 
k8s
$ ssh k8s-master 
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-60-generic x86_64)

• ETCD 툴의 설치 여부 확인

$ etcdctl version
etcdctl version: 3.5.7
API version: 3.5

• ETCD 백업 (https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#backing-up-an-etcd-cluster)

# trusted-ca-file 검색
# trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
$ ps -ef | grep kube | grep trusted-ca-file
... --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

# cert-file 검색
# cert-file=/etc/kubernetes/pki/etcd/server.crt
$ ps -ef | grep kube | grep cert-file
... --cert-file=/etc/kubernetes/pki/etcd/server.crt ...

# key-file 검색
# key-file=/etc/kubernetes/pki/etcd/server.key
$ ps -ef | grep kube | grep key-file
... --key-file=/etc/kubernetes/pki/etcd/server.key ...

# Snapshot using etcdctl options
$ sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /tmp/etcd-backup
# 스냅샷 파일 안에 쿠버네티스 상태 정보 저장된다. (pods, deployments ...)
{"level":"info","ts":"2023-07-10T14:12:12.082+0900","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/tmp/etcd-backup.part"}
{"level":"info","ts":"2023-07-10T14:12:12.101+0900","logger":"client","caller":"v3@v3.5.7/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-07-10T14:12:12.101+0900","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-07-10T14:12:12.267+0900","logger":"client","caller":"v3@v3.5.7/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-07-10T14:12:12.283+0900","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"11 MB","took":"now"}
{"level":"info","ts":"2023-07-10T14:12:12.284+0900","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/tmp/etcd-backup"}
Snapshot saved at /tmp/etcd-backup

• snapshot 파일을 이용해 복원했을 때 원래대로 복원되는지 확인하기 위해 현재 ETCD 상태를 수정 

#deployments를 삭제하여 상태를 변경
$ kubectl get deployments.apps
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
front-end   2/2     2            2           130d
webui       3/3     3            3           130d
$ kubectl delete deployments.apps front-end
deployment.apps "front-end" deleted
$ kubectl get deployments.apps
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
webui   3/3     3            3           130d

• ETCD 복원 (https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#restoring-an-etcd-cluster)

# 기존에 운영중인 공간(/var/lib/etcd)에는 이미 etcd가 작동하고 있기 때문에 
# 새로운 공간(/var/lib/etcd-new)에 restore 해줘야 한다.
$ sudo ETCDCTL_API=3 \
    etcdctl snapshot restore \
    --data-dir /var/lib/etcd-new /tmp/etcd-backup
    
# 데이터 저장소 확인
$ sudo ls /var/lib/etcd
member
$ sudo tree /var/lib/etcd
/var/lib/etcd
└── member
    ├── snap
    │   ├── 0000000000000008-0000000000033466.snap
    │   ├── 0000000000000008-0000000000035b77.snap
    │   ├── 0000000000000008-0000000000038288.snap
    │   ├── 0000000000000008-000000000003a999.snap
    │   ├── 0000000000000008-000000000003d0aa.snap
    │   └── db
    └── wal
        ├── 0.tmp
        ├── 0000000000000000-0000000000000000.wal
        ├── 0000000000000001-00000000000136d4.wal
        ├── 0000000000000002-0000000000027946.wal
        └── 0000000000000003-000000000003db25.wal

3 directories, 11 files

• ETCD Pod에 복원된 etcd-data 위치를 적용하고 Pod 다시 시작

# etcd pod의 data 저장소를 /var/lib/etcd-new로 변경해서 다시실행
$ sudo vi /etc/kubernetes/manifests/etcd.yaml 

# 대문자 G를 입력하여 yaml파일 맨 뒤로 이동
# path를 etcd에서 etcd-new로 변경 후 저장(:wq)하면, 
# static pod는 kublet이라는 데몬이 자동으로 restart해준다.

# etcd.yaml
...
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd-new
      type: DirectoryOrCreate
    name: etcd-data
status: {}

# 복원 확인
$ kubectl get deployments
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
front-end   2/2     2            2           130d
webui       3/3     3            3           130d