Kubernetes 를 쓰면서 발생한 각종 문제에 대한 공유입니다




TroubleShooting





네트워크 (DNS) 문제로 ContainerCreating 에 멈춘 경우.




대략 아래와 같은 현상입니다.



 Warning  FailedCreatePodSandBox  2m (x177 over 7m)  kubelet, test-kube-node001-ncl  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "62ea9854582bbdb08c8b29af9da2499f5146facee60440ee65cb440f47b36c89" network for pod "dst-server-688766c9d5-2m8vl": NetworkPlugin cni failed to set up pod "dst-server-688766c9d5-2m8vl_zepeto" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.1.1/24



cni 가 꼬여서, Cluster 전체가 망가진 상황입니다.


해결책은 Cluster 를 다시 구성하는 수 밖엔 없는것 같습니다. Flannel 을 사용할때 주로 발생하는 문제로 생각됩니다.


# kubeadm reset

# systemctl stop kubelet

# systemctl stop docker

# rm -rf /var/lib/cni/

# rm -rf /var/lib/kubelet/*

# rm -rf /etc/cni/

# ifconfig cni0 down

# ifconfig flannel.1 down

# ifconfig docker0 down

# ip link delete cni0

# ip link delete flannel.1


이후에 /sbin/ifconfig 를 실행해서, 모든 가상 네트워크가 삭제되었음을 확인합니다.

그리고 나서 kubeadm 을 통한 initialize 부터 다시 하시면 됩니다. ㅠㅠ







그 외에...





Kubernetes Install Script




Kubernetes 구성을 위한 ShellScript 입니다.



Master Node & Worker Node 공통.

DockerWithKubernetes_Common.txt



#!/bin/bash


# Check permission

if [ "$EUID" -ne 0 ]

  then echo "Please run as root (SUDO)"

  exit

fi




# update yum

sudo yum update -y



# Stop Firewalld

sudo systemctl stop firewalld

sudo systemctl disable firewalld



# uninstall old version docker

sudo yum remove -y docker \

docker-client \

docker-client-latest \

docker-common \

docker-latest \

docker-latest-logrotate \

docker-logrotate \

docker-selinux \

docker-engine-selinux \

docker-engine



# SETUP THE REPOSITORY


## Install Required Packages

sudo yum install -y yum-utils \

device-mapper-persistent-data \

lvm2



## Setup Repository

sudo yum-config-manager \

--add-repo \

https://download.docker.com/linux/centos/docker-ce.repo




# Install Docker

sudo yum install -y docker-ce



# Start Docker

sudo systemctl start docker

sudo systemctl enable docker



# Add User To Docker Group

sudo /sbin/usermod -aG docker $USER






# INSTALL KUBERNETES

sudo bash -c 'cat <<EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

exclude=kube*

EOF'



# Disable Security Linux

sudo /sbin/setenforce 0


# Disable SWAP

sudo /sbin/swapoff -a


# Set IP Forward

echo 1 > sudo /proc/sys/net/ipv4/ip_forward



# Network Setting

sudo bash -c 'cat <<EOF > /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

EOF'

sudo sysctl --system


# Install Kubernetes

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes



# Start kubelet

sudo systemctl daemon-reload

sudo systemctl restart kubelet

sudo systemctl enable kubelet





Master Node 를 위한 추가 부분


#!/bin/bash



# Check permission

if [ "$EUID" -ne 0 ]

  then echo "Please run as root (SUDO)"

  exit

fi




# Setting Environment

sudo echo 1 > /proc/sys/net/ipv4/ip_forward



# kubeadm init --pod-network-cidr 10.244.0.0/16

sudo kubeadm init --pod-network-cidr 192.168.0.0/16

sudo export KUBECONFIG=/etc/kubernetes/admin.conf



mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config



kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

kubectl get pods --all-namespaces







Kubernetes Uninstall




Kubernetes 삭제하는 방법


kubeadm reset

systemctl stop kubelet

systemctl stop docker

rm -rf /var/lib/cni/

rm -rf /var/lib/kubelet/*

rm -rf /run/flannel

rm -rf /etc/cni/

rm -rf /etc/kubernetes

rm -rf /var/lib/etcd/


ip link delete cni0

ip link delete flannel.1


yum remove -y kubelet

yum remove -y kubectl

yum remove -y kubeadm


systemctl start docker