---
html:
  toc: true
  offline: true
export_on_save:
  html: true
---
# kubernetes 1.13 upgrade
---
3대 서버에 마스터 1개, 노드1, 노도2의 쿠버네티스 클러스터를 구성하기 위한 설치 방법을 설명합니다.
각 서버의 호스트 이름 및 IP 는 아래와 같이 가정합니다. 사용자의 클러스터 환경에 맞게 IP를 수정하시기 바랍니다.
csle1  : 192.168.0.5 (master)
csle2  : 192.168.0.6 (node1)
csle3  : 192.168.0.7 (node2)

## 모든 서버에 공통으로 설치하기

### [모든 서버 공통] 기존 k8s 삭제
```
sudo su -
kubeadm reset
```
```
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
# rm -rf /etc/cni/
rm -rf /run/flannel
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
apt-get purge -y kubeadm kubectl kubelet kubernetes-cni kube*
apt-get autoremove  
sudo rm -rf ~/.kube
systemctl start docker
```

### [모든 서버 공통] kubernetes 최신 버전 install (2019.03.10일 기준 1.13.4)
```
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat << EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update && \
sudo apt-get install -y kubelet=1.13.4-00 kubeadm=1.13.4-00 kubectl=1.13.4-00 kubernetes-cni=0.6.0-00
```

### [모든 서버 공통] docker daemon.json 파일 수정
https://kubernetes.io/docs/setup/cri/
```
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=cgroupfs"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl restart docker
```

### [모든 서버 공통] 쿠버네티스 cgroup 설정 확인
cgroups는 호스트의 자원을 제한해서 각 컨테이너에 할당하는 역할.
docker-ce와 kubernetes가 동일한 제어그룹(cgroup)에 속하는지 확인해야 한다.
docker의 경우 cgroupfs에 속해야 하는데 기본 값은 systemd이다.
# docker info | grep -i cgroup 명령을 통해 확인 가능
```
docker info | grep -i cgroup
Cgroup Driver: cgroupfs
```

### [모든 서버 공통] 10-kubeadm.conf 수정하기
vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
```
Environment="KUBELET_CGROUP_ARGS=-cgroup-driver=cgroupfs"
```
```
systemctl daemon-reload
systemctl restart kubelet
```

### [모든 서버 공통] [optional] resolv.conf 파일 확인
가상환경으로 쿠버네티스를 세팅해서 사용하는 경우, coreDNS가 CrashLoopBackOff상태로 동작하지 않는 경우가 발생한다.
따라서, resolv.conf를 확인해서 필요시 세팅한다.
/etc/resolv.conf 파일내에 nameserver 127.0.1.1만 존재하는 경우 에러가 발생한다.
따라서, 아래와 같이 실행하여야 함.
```
vi /etc/resolvconf/resolv.conf.d/tail
nameserver 8.8.8.8 [추가]
```

```
sudo service resolvconf restart
```

vi /etc/resolv.conf 를 통해 8.8.8.8 주소가 추가되었는지 확인한다.
```
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
nameserver 8.8.8.8
```

## Kubernetes master (csle1) 설정하기

### [Master노드: csle1] kubeadm init
```
sudo su -
swapoff -a
# 위의 서비스 설정을 데몬에 즉시 반영하려면 아래 명령어를 실행하면 됩니다.
systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet
```

```
sudo su -
kubeadm init --apiserver-advertise-address=192.168.0.5 --pod-network-cidr=10.244.0.0/16
```

### [Master: csle1] K8s master 노드 kubectl 실행환경 설정하기
```
sudo su csle
mkdir -p $HOME/.kube
yes | sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=$HOME/.kube/config
echo "export KUBECONFIG=$HOME/.kube/config" | tee -a ~/.bashrc
kubectl get componentstatus
```

## Kubernetes node1, node2 설정하기 (csle2, csle3)

### [csle2, csle3] kubernetes master에 join 하기

```sh
swapoff -a
systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet
```

kubernetes master에서 init시 생성된 script와 추가 옵션을 이용해 master에 join 합니다.

```
kubeadm join 192.168.0.5:6443 --token 77qs8e.czmmpumc6kmi9mua --discovery-token-ca-cert-hash sha256:90125404d068262ce732ced3b097513104413246d636f9434bb730837f50227
```

## kubernetes master에서 확인하기 (csle1)

### [Master: csle1] kubernetes 클러스터 환경 구성을 위한 yaml 환경 설정하기

kubernetes master에서 클러스터 환경 구성을 위한 yaml 파일들을 이용해서 서비스를 생성합니다.

```
sudo su csle
cd /home/csle/ksb-csle/kubernetes/env_set
kubectl create -f kube-flannel.yml
kubectl create -f kubernetes-dashboard.yaml
kubectl create -f kubernetes-dashboard-admin-rbac.yaml
kubectl create -f ServiceAccount.yml
kubectl create -f ClusterRoleBinding.yml
kubectl create -f k8s-heapster/
```
```
cd /home/csle/ksb-csle/kubernetes/env_set
kubectl delete -f kube-flannel.yml
kubectl delete -f kubernetes-dashboard.yaml
kubectl delete -f kubernetes-dashboard-admin-rbac.yaml
kubectl delete -f ServiceAccount.yml
kubectl delete -f ClusterRoleBinding.yml
kubectl delete -f k8s-heapster/
```
클러스터 환경 구성을 위한 pod가 정상적으로 구동되었는지 확인합니다.

```
csle@csle1:~/ksb-csle/kubernetes/env_set$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
kube-system   etcd-csle1                              1/1       Running   0          6m
kube-system   heapster-dfd674df9-nttw8                1/1       Running   0          1m
kube-system   kube-apiserver-csle1                    1/1       Running   0          6m
kube-system   kube-controller-manager-csle1           1/1       Running   0          5m
kube-system   kube-dns-6f4fd4bdf-g6lh6                3/3       Running   0          6m
kube-system   kube-flannel-ds-amd64-8wgn7             1/1       Running   0          1m
kube-system   kube-flannel-ds-amd64-h49xx             1/1       Running   0          1m
kube-system   kube-flannel-ds-amd64-jf96q             1/1       Running   0          1m
kube-system   kube-flannel-ds-amd64-w6lsc             1/1       Running   0          1m
kube-system   kube-proxy-4mxf6                        1/1       Running   0          6m
kube-system   kube-proxy-4qfb2                        1/1       Running   0          5m
kube-system   kube-proxy-6qpzp                        1/1       Running   0          5m
kube-system   kube-proxy-w5bsw                        1/1       Running   0          5m
kube-system   kube-scheduler-csle1                    1/1       Running   0          7m
kube-system   kubernetes-dashboard-6c664cf6c5-qjfm6   1/1       Running   0          1m
kube-system   monitoring-grafana-76848b566c-lqtnj     1/1       Running   0          1m
kube-system   monitoring-influxdb-6c4b84d695-7qc4g    1/1       Running   0          1m
```

### [Master: csle1] Ingress 환경 설정하기
kubernetes master에서 ingress를 설정합니다.

```sh
cd /home/csle/ksb-csle/kubernetes/env_set
kubectl create -f default-backend-for-ingress.yaml
kubectl create -f configmap-nginx-ingress-controller.yaml
kubectl create -f deploy-nginx-ingress-controller.yaml
kubectl create -f ingress-rule.yaml
kubectl create -f svc-expose-by-nodeport.yaml
```
##### 삭제 방법
```
kubectl delete -f default-backend-for-ingress.yaml
kubectl delete -f configmap-nginx-ingress-controller.yaml
kubectl delete -f deploy-nginx-ingress-controller.yaml
kubectl delete -f ingress-rule.yaml
kubectl delete -f svc-expose-by-nodeport.yaml
```

```sh
csle@csle1:~/ksb-csle/kubernetes/env_set$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                        READY     STATUS    RESTARTS   AGE
default       default-http-backend-66fbbd8844-2x96j       1/1       Running   0          27s
default       default-http-backend-66fbbd8844-x67j5       1/1       Running   0          27s
default       nginx-ingress-controller-6857676bb9-bspbm   1/1       Running   0          26s
default       nginx-ingress-controller-6857676bb9-jhg4t   0/1       Running   0          26s
kube-system   etcd-csle1                                  1/1       Running   0          7m
kube-system   heapster-dfd674df9-nttw8                    1/1       Running   0          3m
kube-system   kube-apiserver-csle1                        1/1       Running   0          7m
kube-system   kube-controller-manager-csle1               1/1       Running   0          7m
kube-system   kube-dns-6f4fd4bdf-g6lh6                    3/3       Running   0          8m
kube-system   kube-flannel-ds-amd64-8wgn7                 1/1       Running   0          3m
kube-system   kube-flannel-ds-amd64-h49xx                 1/1       Running   0          3m
kube-system   kube-flannel-ds-amd64-jf96q                 1/1       Running   0          3m
kube-system   kube-flannel-ds-amd64-w6lsc                 1/1       Running   0          3m
kube-system   kube-proxy-4mxf6                            1/1       Running   0          8m
kube-system   kube-proxy-4qfb2                            1/1       Running   0          6m
kube-system   kube-proxy-6qpzp                            1/1       Running   0          7m
kube-system   kube-proxy-w5bsw                            1/1       Running   0          7m
kube-system   kube-scheduler-csle1                        1/1       Running   0          8m
kube-system   kubernetes-dashboard-6c664cf6c5-qjfm6       1/1       Running   0          3m
kube-system   monitoring-grafana-76848b566c-lqtnj         1/1       Running   0          3m
kube-system   monitoring-influxdb-6c4b84d695-7qc4g        1/1       Running   0          3m
```

```
csle@csle1:~/ksb-csle/kubernetes/env_set$ k get nodes
NAME    STATUS   ROLES    AGE     VERSION
csle1   Ready    master   5m3s    v1.13.4
csle2   Ready    <none>   3m20s   v1.13.4
csle3   Ready    <none>   3m10s   v1.13.4
csle4   Ready    <none>   2m17s   v1.13.4
```

```
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
kubectl proxy --port=9999 --address='192.168.0.5' --accept-hosts="^*$" &
```

### [Master: csle1] Dashboard URL에 접속하기
http://192.168.0.5:9999/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

# FAQ
## [ERROR] coredns stuck at ContainerCreating
### kubernetes 홈페이지의 FAQ 참조
https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/

업그레드를 하고 coredns가 ContainerCreating되는 경우,
```
csle@csle1:~/ksb-csle/kubernetes/env_set$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY   STATUS              RESTARTS   AGE
kube-system   coredns-86c58d9df4-8vtpz                0/1     ContainerCreating   0          23m
kube-system   coredns-86c58d9df4-w46t5                1/1     Running             0          23m
kube-system   etcd-csle1                              1/1     Running             0          22m
kube-system   heapster-6cc9489ff-nfw2g                1/1     Running             0          119s
kube-system   kube-apiserver-csle1                    1/1     Running             0          22m
kube-system   kube-controller-manager-csle1           1/1     Running             0          22m
kube-system   kube-flannel-ds-amd64-htfdm             1/1     Running             0          2m3s
kube-system   kube-flannel-ds-amd64-hwt4q             1/1     Running             0          2m3s
kube-system   kube-flannel-ds-amd64-m74ll             1/1     Running             0          2m3s
kube-system   kube-flannel-ds-amd64-p7lds             1/1     Running             0          2m3s
kube-system   kube-proxy-bzjvj                        1/1     Running             0          9m21s
kube-system   kube-proxy-gskr7                        1/1     Running             0          22m
kube-system   kube-proxy-np85l                        1/1     Running             0          9m11s
kube-system   kube-proxy-zvnf4                        1/1     Running             0          23m
kube-system   kube-scheduler-csle1                    1/1     Running             0          22m
kube-system   kubernetes-dashboard-79ff88449c-x6vn7   1/1     Running             0          2m3s
kube-system   monitoring-grafana-b575bb8ff-zv6bm      1/1     Running             0          2m
kube-system   monitoring-influxdb-6d65f866f5-hrbg7    1/1     Running             0          2m
```
kubectl describe po coredns-86c58d9df4-8vtpz -n kube-system
```
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Warning  FailedScheduling        35m (x2 over 35m)      default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling        22m (x17 over 35m)     default-scheduler  0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling        21m (x4 over 21m)      default-scheduler  0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling        15m (x18 over 21m)     default-scheduler  0/4 nodes are available: 4 node(s) had taints that the pod didn't tolerate.
  Warning  FailedCreatePodSandBox  14m                    kubelet, csle4     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e3d5c11d149862e9363fe99f6e9c13437940b9e5b3ec7b2f067217703ec25667" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: could not add IP address to "cni0": file exists
  Warning  FailedCreatePodSandBox  14m                    kubelet, csle4     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6acb551f701e76652d6d1e4fdb061ba9e28daaa7f079d3c88c318487463d4ac8" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
  Warning  FailedCreatePodSandBox  14m                    kubelet, csle4     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b260044185b610f182bd124fa09e280b75ba3bd7ca2b0bcfb312f85c4b0aab9c" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
  Warning  FailedCreatePodSandBox  14m                    kubelet, csle4     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "398151b2054f7568246fad677992018367d086e192ff5418aacffa34bf242b04" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
  Warning  FailedCreatePodSandBox  14m                    kubelet, csle4     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bdb5f14dd25285df5e59d0eebb9784952fbf3d776d2c8050ee72344a0028259c" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
  Warning  FailedCreatePodSandBox  14m                    kubelet, csle4     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f65c994064131557062e70b17569f51aebb5494d97002281ddbf5296e5f543cd" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
  Warning  FailedCreatePodSandBox  14m                    kubelet, csle4     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b65fd62d9544d5f713f98d40d5089421e7c206eba1ed5e19c8e56cb65e32a74d" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
  Warning  FailedCreatePodSandBox  14m                    kubelet, csle4     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "caf0985a7467322508e3447f35a02985692e944969cbfe082899cfa0ea0c8cae" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
  Warning  FailedCreatePodSandBox  14m                    kubelet, csle4     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4c771f07e241ae1564854c65f48c53afa6166e3231fbc4c356b70753cba33585" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
  Normal   SandboxChanged          14m (x12 over 14m)     kubelet, csle4     Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  4m28s (x266 over 14m)  kubelet, csle4     (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "01c5bc76700598fb67bd2fda4694f1992556b73afb9389955639ec1b2c08dda7" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
```

### [해결방법]

https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#coredns-pods-have-crashloopbackoff-or-error-state

https://coredns.io/plugins/loop/#troubleshooting

/etc/resolv.conf 파일내에 nameserver 127.0.1.1 만 존재하는 경우 발생한다.
따라서, 아래와 같이 실행하여야 함.

```
vi /etc/resolvconf/resolv.conf.d/tail
nameserver 8.8.8.8
```

```
sudo service resolvconf restart
```
vi /etc/resolv.conf

```
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
nameserver 8.8.8.8
```