Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
---
html:
toc: true
offline: true
export_on_save:
html: true
---
# kubernetes 1.13 upgrade
---
3대 서버에 마스터 1개, 노드1, 노도2의 쿠버네티스 클러스터를 구성하기 위한 설치 방법을 설명합니다.
각 서버의 호스트 이름 및 IP 는 아래와 같이 가정합니다. 사용자의 클러스터 환경에 맞게 IP를 수정하시기 바랍니다.
csle1 : 192.168.0.5 (master)
csle2 : 192.168.0.6 (node1)
csle3 : 192.168.0.7 (node2)
## 모든 서버에 공통으로 설치하기
### [모든 서버 공통] 기존 k8s 삭제
```
sudo su -
kubeadm reset
```
```
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
# rm -rf /etc/cni/
rm -rf /run/flannel
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
apt-get purge -y kubeadm kubectl kubelet kubernetes-cni kube*
apt-get autoremove
sudo rm -rf ~/.kube
systemctl start docker
```
### [모든 서버 공통] kubernetes 최신 버전 install (2019.03.10일 기준 1.13.4)
```
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat << EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update && \
sudo apt-get install -y kubelet=1.13.4-00 kubeadm=1.13.4-00 kubectl=1.13.4-00 kubernetes-cni=0.6.0-00
```
### [모든 서버 공통] docker daemon.json 파일 수정
https://kubernetes.io/docs/setup/cri/
```
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=cgroupfs"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl restart docker
```
### [모든 서버 공통] 쿠버네티스 cgroup 설정 확인
cgroups는 호스트의 자원을 제한해서 각 컨테이너에 할당하는 역할.
docker-ce와 kubernetes가 동일한 제어그룹(cgroup)에 속하는지 확인해야 한다.
docker의 경우 cgroupfs에 속해야 하는데 기본 값은 systemd이다.
# docker info | grep -i cgroup 명령을 통해 확인 가능
```
docker info | grep -i cgroup
Cgroup Driver: cgroupfs
```
### [모든 서버 공통] 10-kubeadm.conf 수정하기
vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
```
Environment="KUBELET_CGROUP_ARGS=-cgroup-driver=cgroupfs"
```
```
systemctl daemon-reload
systemctl restart kubelet
```
### [모든 서버 공통] [optional] resolv.conf 파일 확인
가상환경으로 쿠버네티스를 세팅해서 사용하는 경우, coreDNS가 CrashLoopBackOff상태로 동작하지 않는 경우가 발생한다.
따라서, resolv.conf를 확인해서 필요시 세팅한다.
/etc/resolv.conf 파일내에 nameserver 127.0.1.1만 존재하는 경우 에러가 발생한다.
따라서, 아래와 같이 실행하여야 함.
```
vi /etc/resolvconf/resolv.conf.d/tail
nameserver 8.8.8.8 [추가]
```
```
sudo service resolvconf restart
```
vi /etc/resolv.conf 를 통해 8.8.8.8 주소가 추가되었는지 확인한다.
```
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
nameserver 8.8.8.8
```
## Kubernetes master (csle1) 설정하기
### [Master노드: csle1] kubeadm init
```
sudo su -
swapoff -a
# 위의 서비스 설정을 데몬에 즉시 반영하려면 아래 명령어를 실행하면 됩니다.
systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet
```
```
sudo su -
kubeadm init --apiserver-advertise-address=192.168.0.5 --pod-network-cidr=10.244.0.0/16
```
### [Master: csle1] K8s master 노드 kubectl 실행환경 설정하기
```
sudo su csle
mkdir -p $HOME/.kube
yes | sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=$HOME/.kube/config
echo "export KUBECONFIG=$HOME/.kube/config" | tee -a ~/.bashrc
kubectl get componentstatus
```
## Kubernetes node1, node2 설정하기 (csle2, csle3)
### [csle2, csle3] kubernetes master에 join 하기
```sh
swapoff -a
systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet
```
kubernetes master에서 init시 생성된 script와 추가 옵션을 이용해 master에 join 합니다.
```
kubeadm join 192.168.0.5:6443 --token 77qs8e.czmmpumc6kmi9mua --discovery-token-ca-cert-hash sha256:90125404d068262ce732ced3b097513104413246d636f9434bb730837f50227
```
## kubernetes master에서 확인하기 (csle1)
### [Master: csle1] kubernetes 클러스터 환경 구성을 위한 yaml 환경 설정하기
kubernetes master에서 클러스터 환경 구성을 위한 yaml 파일들을 이용해서 서비스를 생성합니다.
```
sudo su csle
cd /home/csle/ksb-csle/kubernetes/env_set
kubectl create -f kube-flannel.yml
kubectl create -f kubernetes-dashboard.yaml
kubectl create -f kubernetes-dashboard-admin-rbac.yaml
kubectl create -f ServiceAccount.yml
kubectl create -f ClusterRoleBinding.yml
kubectl create -f k8s-heapster/
```
```
cd /home/csle/ksb-csle/kubernetes/env_set
kubectl delete -f kube-flannel.yml
kubectl delete -f kubernetes-dashboard.yaml
kubectl delete -f kubernetes-dashboard-admin-rbac.yaml
kubectl delete -f ServiceAccount.yml
kubectl delete -f ClusterRoleBinding.yml
kubectl delete -f k8s-heapster/
```
클러스터 환경 구성을 위한 pod가 정상적으로 구동되었는지 확인합니다.
```
csle@csle1:~/ksb-csle/kubernetes/env_set$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-csle1 1/1 Running 0 6m
kube-system heapster-dfd674df9-nttw8 1/1 Running 0 1m
kube-system kube-apiserver-csle1 1/1 Running 0 6m
kube-system kube-controller-manager-csle1 1/1 Running 0 5m
kube-system kube-dns-6f4fd4bdf-g6lh6 3/3 Running 0 6m
kube-system kube-flannel-ds-amd64-8wgn7 1/1 Running 0 1m
kube-system kube-flannel-ds-amd64-h49xx 1/1 Running 0 1m
kube-system kube-flannel-ds-amd64-jf96q 1/1 Running 0 1m
kube-system kube-flannel-ds-amd64-w6lsc 1/1 Running 0 1m
kube-system kube-proxy-4mxf6 1/1 Running 0 6m
kube-system kube-proxy-4qfb2 1/1 Running 0 5m
kube-system kube-proxy-6qpzp 1/1 Running 0 5m
kube-system kube-proxy-w5bsw 1/1 Running 0 5m
kube-system kube-scheduler-csle1 1/1 Running 0 7m
kube-system kubernetes-dashboard-6c664cf6c5-qjfm6 1/1 Running 0 1m
kube-system monitoring-grafana-76848b566c-lqtnj 1/1 Running 0 1m
kube-system monitoring-influxdb-6c4b84d695-7qc4g 1/1 Running 0 1m
```
### [Master: csle1] Ingress 환경 설정하기
kubernetes master에서 ingress를 설정합니다.
```sh
cd /home/csle/ksb-csle/kubernetes/env_set
kubectl create -f default-backend-for-ingress.yaml
kubectl create -f configmap-nginx-ingress-controller.yaml
kubectl create -f deploy-nginx-ingress-controller.yaml
kubectl create -f ingress-rule.yaml
kubectl create -f svc-expose-by-nodeport.yaml
```
##### 삭제 방법
```
kubectl delete -f default-backend-for-ingress.yaml
kubectl delete -f configmap-nginx-ingress-controller.yaml
kubectl delete -f deploy-nginx-ingress-controller.yaml
kubectl delete -f ingress-rule.yaml
kubectl delete -f svc-expose-by-nodeport.yaml
```
```sh
csle@csle1:~/ksb-csle/kubernetes/env_set$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default default-http-backend-66fbbd8844-2x96j 1/1 Running 0 27s
default default-http-backend-66fbbd8844-x67j5 1/1 Running 0 27s
default nginx-ingress-controller-6857676bb9-bspbm 1/1 Running 0 26s
default nginx-ingress-controller-6857676bb9-jhg4t 0/1 Running 0 26s
kube-system etcd-csle1 1/1 Running 0 7m
kube-system heapster-dfd674df9-nttw8 1/1 Running 0 3m
kube-system kube-apiserver-csle1 1/1 Running 0 7m
kube-system kube-controller-manager-csle1 1/1 Running 0 7m
kube-system kube-dns-6f4fd4bdf-g6lh6 3/3 Running 0 8m
kube-system kube-flannel-ds-amd64-8wgn7 1/1 Running 0 3m
kube-system kube-flannel-ds-amd64-h49xx 1/1 Running 0 3m
kube-system kube-flannel-ds-amd64-jf96q 1/1 Running 0 3m
kube-system kube-flannel-ds-amd64-w6lsc 1/1 Running 0 3m
kube-system kube-proxy-4mxf6 1/1 Running 0 8m
kube-system kube-proxy-4qfb2 1/1 Running 0 6m
kube-system kube-proxy-6qpzp 1/1 Running 0 7m
kube-system kube-proxy-w5bsw 1/1 Running 0 7m
kube-system kube-scheduler-csle1 1/1 Running 0 8m
kube-system kubernetes-dashboard-6c664cf6c5-qjfm6 1/1 Running 0 3m
kube-system monitoring-grafana-76848b566c-lqtnj 1/1 Running 0 3m
kube-system monitoring-influxdb-6c4b84d695-7qc4g 1/1 Running 0 3m
```
```
csle@csle1:~/ksb-csle/kubernetes/env_set$ k get nodes
NAME STATUS ROLES AGE VERSION
csle1 Ready master 5m3s v1.13.4
csle2 Ready <none> 3m20s v1.13.4
csle3 Ready <none> 3m10s v1.13.4
csle4 Ready <none> 2m17s v1.13.4
```
```
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
kubectl proxy --port=9999 --address='192.168.0.5' --accept-hosts="^*$" &
```
### [Master: csle1] Dashboard URL에 접속하기
http://192.168.0.5:9999/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
# FAQ
## [ERROR] coredns stuck at ContainerCreating
### kubernetes 홈페이지의 FAQ 참조
https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/
업그레드를 하고 coredns가 ContainerCreating되는 경우,
```
csle@csle1:~/ksb-csle/kubernetes/env_set$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-86c58d9df4-8vtpz 0/1 ContainerCreating 0 23m
kube-system coredns-86c58d9df4-w46t5 1/1 Running 0 23m
kube-system etcd-csle1 1/1 Running 0 22m
kube-system heapster-6cc9489ff-nfw2g 1/1 Running 0 119s
kube-system kube-apiserver-csle1 1/1 Running 0 22m
kube-system kube-controller-manager-csle1 1/1 Running 0 22m
kube-system kube-flannel-ds-amd64-htfdm 1/1 Running 0 2m3s
kube-system kube-flannel-ds-amd64-hwt4q 1/1 Running 0 2m3s
kube-system kube-flannel-ds-amd64-m74ll 1/1 Running 0 2m3s
kube-system kube-flannel-ds-amd64-p7lds 1/1 Running 0 2m3s
kube-system kube-proxy-bzjvj 1/1 Running 0 9m21s
kube-system kube-proxy-gskr7 1/1 Running 0 22m
kube-system kube-proxy-np85l 1/1 Running 0 9m11s
kube-system kube-proxy-zvnf4 1/1 Running 0 23m
kube-system kube-scheduler-csle1 1/1 Running 0 22m
kube-system kubernetes-dashboard-79ff88449c-x6vn7 1/1 Running 0 2m3s
kube-system monitoring-grafana-b575bb8ff-zv6bm 1/1 Running 0 2m
kube-system monitoring-influxdb-6d65f866f5-hrbg7 1/1 Running 0 2m
```
kubectl describe po coredns-86c58d9df4-8vtpz -n kube-system
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 35m (x2 over 35m) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
Warning FailedScheduling 22m (x17 over 35m) default-scheduler 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
Warning FailedScheduling 21m (x4 over 21m) default-scheduler 0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.
Warning FailedScheduling 15m (x18 over 21m) default-scheduler 0/4 nodes are available: 4 node(s) had taints that the pod didn't tolerate.
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e3d5c11d149862e9363fe99f6e9c13437940b9e5b3ec7b2f067217703ec25667" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: could not add IP address to "cni0": file exists
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6acb551f701e76652d6d1e4fdb061ba9e28daaa7f079d3c88c318487463d4ac8" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b260044185b610f182bd124fa09e280b75ba3bd7ca2b0bcfb312f85c4b0aab9c" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "398151b2054f7568246fad677992018367d086e192ff5418aacffa34bf242b04" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bdb5f14dd25285df5e59d0eebb9784952fbf3d776d2c8050ee72344a0028259c" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f65c994064131557062e70b17569f51aebb5494d97002281ddbf5296e5f543cd" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b65fd62d9544d5f713f98d40d5089421e7c206eba1ed5e19c8e56cb65e32a74d" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "caf0985a7467322508e3447f35a02985692e944969cbfe082899cfa0ea0c8cae" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4c771f07e241ae1564854c65f48c53afa6166e3231fbc4c356b70753cba33585" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Normal SandboxChanged 14m (x12 over 14m) kubelet, csle4 Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 4m28s (x266 over 14m) kubelet, csle4 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "01c5bc76700598fb67bd2fda4694f1992556b73afb9389955639ec1b2c08dda7" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
```
### [해결방법]
https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#coredns-pods-have-crashloopbackoff-or-error-state
https://coredns.io/plugins/loop/#troubleshooting
/etc/resolv.conf 파일내에 nameserver 127.0.1.1 만 존재하는 경우 발생한다.
따라서, 아래와 같이 실행하여야 함.
```
vi /etc/resolvconf/resolv.conf.d/tail
nameserver 8.8.8.8
```
```
sudo service resolvconf restart
```
vi /etc/resolv.conf
```
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
nameserver 8.8.8.8
```