3대 서버에 마스터 1개, 노드1, 노도2의 쿠버네티스 클러스터를 구성하기 위한 설치 방법을 설명합니다.
각 서버의 호스트 이름 및 IP 는 아래와 같이 가정합니다. 사용자의 클러스터 환경에 맞게 IP를 수정하시기 바랍니다.
csle1 : 192.168.0.5 (master)
csle2 : 192.168.0.6 (node1)
csle3 : 192.168.0.7 (node2)
sudo su -
kubeadm reset
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
# rm -rf /etc/cni/
rm -rf /run/flannel
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
apt-get purge -y kubeadm kubectl kubelet kubernetes-cni kube*
apt-get autoremove
sudo rm -rf ~/.kube
systemctl start docker
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat << EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update && \
sudo apt-get install -y kubelet=1.13.4-00 kubeadm=1.13.4-00 kubectl=1.13.4-00 kubernetes-cni=0.6.0-00
https://kubernetes.io/docs/setup/cri/
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=cgroupfs"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl restart docker
cgroups는 호스트의 자원을 제한해서 각 컨테이너에 할당하는 역할.
docker-ce와 kubernetes가 동일한 제어그룹(cgroup)에 속하는지 확인해야 한다.
docker의 경우 cgroupfs에 속해야 하는데 기본 값은 systemd이다.
docker info | grep -i cgroup
Cgroup Driver: cgroupfs
vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_CGROUP_ARGS=-cgroup-driver=cgroupfs"
systemctl daemon-reload
systemctl restart kubelet
가상환경으로 쿠버네티스를 세팅해서 사용하는 경우, coreDNS가 CrashLoopBackOff상태로 동작하지 않는 경우가 발생한다.
따라서, resolv.conf를 확인해서 필요시 세팅한다.
/etc/resolv.conf 파일내에 nameserver 127.0.1.1만 존재하는 경우 에러가 발생한다.
따라서, 아래와 같이 실행하여야 함.
vi /etc/resolvconf/resolv.conf.d/tail
nameserver 8.8.8.8 [추가]
sudo service resolvconf restart
vi /etc/resolv.conf 를 통해 8.8.8.8 주소가 추가되었는지 확인한다.
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
nameserver 8.8.8.8
sudo su -
swapoff -a
# 위의 서비스 설정을 데몬에 즉시 반영하려면 아래 명령어를 실행하면 됩니다.
systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet
sudo su -
kubeadm init --apiserver-advertise-address=192.168.0.5 --pod-network-cidr=10.244.0.0/16
sudo su csle
mkdir -p $HOME/.kube
yes | sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=$HOME/.kube/config
echo "export KUBECONFIG=$HOME/.kube/config" | tee -a ~/.bashrc
kubectl get componentstatus
swapoff -a systemctl daemon-reload systemctl restart kubelet systemctl status kubelet
kubernetes master에서 init시 생성된 script와 추가 옵션을 이용해 master에 join 합니다.
kubeadm join 192.168.0.5:6443 --token 77qs8e.czmmpumc6kmi9mua --discovery-token-ca-cert-hash sha256:90125404d068262ce732ced3b097513104413246d636f9434bb730837f50227
kubernetes master에서 클러스터 환경 구성을 위한 yaml 파일들을 이용해서 서비스를 생성합니다.
sudo su csle
cd /home/csle/ksb-csle/kubernetes/env_set
kubectl create -f kube-flannel.yml
kubectl create -f kubernetes-dashboard.yaml
kubectl create -f kubernetes-dashboard-admin-rbac.yaml
kubectl create -f ServiceAccount.yml
kubectl create -f ClusterRoleBinding.yml
kubectl create -f k8s-heapster/
cd /home/csle/ksb-csle/kubernetes/env_set
kubectl delete -f kube-flannel.yml
kubectl delete -f kubernetes-dashboard.yaml
kubectl delete -f kubernetes-dashboard-admin-rbac.yaml
kubectl delete -f ServiceAccount.yml
kubectl delete -f ClusterRoleBinding.yml
kubectl delete -f k8s-heapster/
클러스터 환경 구성을 위한 pod가 정상적으로 구동되었는지 확인합니다.
csle@csle1:~/ksb-csle/kubernetes/env_set$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-csle1 1/1 Running 0 6m
kube-system heapster-dfd674df9-nttw8 1/1 Running 0 1m
kube-system kube-apiserver-csle1 1/1 Running 0 6m
kube-system kube-controller-manager-csle1 1/1 Running 0 5m
kube-system kube-dns-6f4fd4bdf-g6lh6 3/3 Running 0 6m
kube-system kube-flannel-ds-amd64-8wgn7 1/1 Running 0 1m
kube-system kube-flannel-ds-amd64-h49xx 1/1 Running 0 1m
kube-system kube-flannel-ds-amd64-jf96q 1/1 Running 0 1m
kube-system kube-flannel-ds-amd64-w6lsc 1/1 Running 0 1m
kube-system kube-proxy-4mxf6 1/1 Running 0 6m
kube-system kube-proxy-4qfb2 1/1 Running 0 5m
kube-system kube-proxy-6qpzp 1/1 Running 0 5m
kube-system kube-proxy-w5bsw 1/1 Running 0 5m
kube-system kube-scheduler-csle1 1/1 Running 0 7m
kube-system kubernetes-dashboard-6c664cf6c5-qjfm6 1/1 Running 0 1m
kube-system monitoring-grafana-76848b566c-lqtnj 1/1 Running 0 1m
kube-system monitoring-influxdb-6c4b84d695-7qc4g 1/1 Running 0 1m
kubernetes master에서 ingress를 설정합니다.
cd /home/csle/ksb-csle/kubernetes/env_set
kubectl create -f default-backend-for-ingress.yaml
kubectl create -f configmap-nginx-ingress-controller.yaml
kubectl create -f deploy-nginx-ingress-controller.yaml
kubectl create -f ingress-rule.yaml
kubectl create -f svc-expose-by-nodeport.yaml
kubectl delete -f default-backend-for-ingress.yaml
kubectl delete -f configmap-nginx-ingress-controller.yaml
kubectl delete -f deploy-nginx-ingress-controller.yaml
kubectl delete -f ingress-rule.yaml
kubectl delete -f svc-expose-by-nodeport.yaml
csle@csle1:~/ksb-csle/kubernetes/env_set$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default default-http-backend-66fbbd8844-2x96j 1/1 Running 0 27s default default-http-backend-66fbbd8844-x67j5 1/1 Running 0 27s default nginx-ingress-controller-6857676bb9-bspbm 1/1 Running 0 26s default nginx-ingress-controller-6857676bb9-jhg4t 0/1 Running 0 26s kube-system etcd-csle1 1/1 Running 0 7m kube-system heapster-dfd674df9-nttw8 1/1 Running 0 3m kube-system kube-apiserver-csle1 1/1 Running 0 7m kube-system kube-controller-manager-csle1 1/1 Running 0 7m kube-system kube-dns-6f4fd4bdf-g6lh6 3/3 Running 0 8m kube-system kube-flannel-ds-amd64-8wgn7 1/1 Running 0 3m kube-system kube-flannel-ds-amd64-h49xx 1/1 Running 0 3m kube-system kube-flannel-ds-amd64-jf96q 1/1 Running 0 3m kube-system kube-flannel-ds-amd64-w6lsc 1/1 Running 0 3m kube-system kube-proxy-4mxf6 1/1 Running 0 8m kube-system kube-proxy-4qfb2 1/1 Running 0 6m kube-system kube-proxy-6qpzp 1/1 Running 0 7m kube-system kube-proxy-w5bsw 1/1 Running 0 7m kube-system kube-scheduler-csle1 1/1 Running 0 8m kube-system kubernetes-dashboard-6c664cf6c5-qjfm6 1/1 Running 0 3m kube-system monitoring-grafana-76848b566c-lqtnj 1/1 Running 0 3m kube-system monitoring-influxdb-6c4b84d695-7qc4g 1/1 Running 0 3m
csle@csle1:~/ksb-csle/kubernetes/env_set$ k get nodes
NAME STATUS ROLES AGE VERSION
csle1 Ready master 5m3s v1.13.4
csle2 Ready <none> 3m20s v1.13.4
csle3 Ready <none> 3m10s v1.13.4
csle4 Ready <none> 2m17s v1.13.4
kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
kubectl proxy --port=9999 --address='192.168.0.5' --accept-hosts="^*$" &
http://192.168.0.5:9999/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/
업그레드를 하고 coredns가 ContainerCreating되는 경우,
csle@csle1:~/ksb-csle/kubernetes/env_set$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-86c58d9df4-8vtpz 0/1 ContainerCreating 0 23m
kube-system coredns-86c58d9df4-w46t5 1/1 Running 0 23m
kube-system etcd-csle1 1/1 Running 0 22m
kube-system heapster-6cc9489ff-nfw2g 1/1 Running 0 119s
kube-system kube-apiserver-csle1 1/1 Running 0 22m
kube-system kube-controller-manager-csle1 1/1 Running 0 22m
kube-system kube-flannel-ds-amd64-htfdm 1/1 Running 0 2m3s
kube-system kube-flannel-ds-amd64-hwt4q 1/1 Running 0 2m3s
kube-system kube-flannel-ds-amd64-m74ll 1/1 Running 0 2m3s
kube-system kube-flannel-ds-amd64-p7lds 1/1 Running 0 2m3s
kube-system kube-proxy-bzjvj 1/1 Running 0 9m21s
kube-system kube-proxy-gskr7 1/1 Running 0 22m
kube-system kube-proxy-np85l 1/1 Running 0 9m11s
kube-system kube-proxy-zvnf4 1/1 Running 0 23m
kube-system kube-scheduler-csle1 1/1 Running 0 22m
kube-system kubernetes-dashboard-79ff88449c-x6vn7 1/1 Running 0 2m3s
kube-system monitoring-grafana-b575bb8ff-zv6bm 1/1 Running 0 2m
kube-system monitoring-influxdb-6d65f866f5-hrbg7 1/1 Running 0 2m
kubectl describe po coredns-86c58d9df4-8vtpz -n kube-system
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 35m (x2 over 35m) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
Warning FailedScheduling 22m (x17 over 35m) default-scheduler 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
Warning FailedScheduling 21m (x4 over 21m) default-scheduler 0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.
Warning FailedScheduling 15m (x18 over 21m) default-scheduler 0/4 nodes are available: 4 node(s) had taints that the pod didn't tolerate.
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e3d5c11d149862e9363fe99f6e9c13437940b9e5b3ec7b2f067217703ec25667" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: could not add IP address to "cni0": file exists
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6acb551f701e76652d6d1e4fdb061ba9e28daaa7f079d3c88c318487463d4ac8" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b260044185b610f182bd124fa09e280b75ba3bd7ca2b0bcfb312f85c4b0aab9c" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "398151b2054f7568246fad677992018367d086e192ff5418aacffa34bf242b04" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bdb5f14dd25285df5e59d0eebb9784952fbf3d776d2c8050ee72344a0028259c" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "f65c994064131557062e70b17569f51aebb5494d97002281ddbf5296e5f543cd" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b65fd62d9544d5f713f98d40d5089421e7c206eba1ed5e19c8e56cb65e32a74d" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "caf0985a7467322508e3447f35a02985692e944969cbfe082899cfa0ea0c8cae" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Warning FailedCreatePodSandBox 14m kubelet, csle4 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4c771f07e241ae1564854c65f48c53afa6166e3231fbc4c356b70753cba33585" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
Normal SandboxChanged 14m (x12 over 14m) kubelet, csle4 Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 4m28s (x266 over 14m) kubelet, csle4 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "01c5bc76700598fb67bd2fda4694f1992556b73afb9389955639ec1b2c08dda7" network for pod "coredns-86c58d9df4-8vtpz": NetworkPlugin cni failed to set up pod "coredns-86c58d9df4-8vtpz_kube-system" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.3.1/24
https://coredns.io/plugins/loop/#troubleshooting
/etc/resolv.conf 파일내에 nameserver 127.0.1.1 만 존재하는 경우 발생한다.
따라서, 아래와 같이 실행하여야 함.
vi /etc/resolvconf/resolv.conf.d/tail
nameserver 8.8.8.8
sudo service resolvconf restart
vi /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
nameserver 8.8.8.8