——————————- Deploy CNI network components—————- —————
———- Deploy flannel ———-
Pod network communication in K8S:
●Communication between containers within a Pod
Containers in the same Pod (containers in a Pod will not cross hosts) share the same network namespace, which is equivalent to them being on the same machine. They can use the localhost address to access each other’s ports.
●Communication between Pods in the same Node
Each Pod has a real global IP address. Different Pods in the same Node can directly use the IP address of the other Pod to communicate. Pod1 and Pod2 are connected to the same docker0/cni0 bridge through Veth. The segments are identical, so they can communicate directly with each other.
●Communication between Pods on different Nodes
The Pod address is in the same network segment as docker0. The docker0 network segment and the host network card are two different network segments, and communication between different Nodes can only be carried out through the host’s physical network card.
To achieve communication between Pods on different Nodes, you must find a way to address and communicate through the host’s physical network card IP address. Therefore, two conditions must be met: the IP of the Pod cannot conflict; the IP of the Pod must be associated with the IP of the Node where it is located. Through this association, Pods on different Nodes can communicate directly through the intranet IP address.
Overlay Network：
Overlay network is a virtual network technology model superimposed on a layer 2 or layer 3 basic network. Hosts in the network are connected through virtual link tunnels.
Through Overlay technology (which can be understood as tunnel technology), a layer of Layer 4 protocol (UDP protocol) is wrapped outside the original message and routed and forwarded through the host network. This method has a certain performance loss, which is mainly reflected in the modification of the original message. Currently Overlay mainly uses VXLAN.
VXLAN:
The source data packet is encapsulated into UDP, and the IP/MAC of the underlying network is used as the outer packet header for encapsulation, and then transmitted over the Ethernet. After reaching the destination, the tunnel endpoint decapsulates and sends the data to the target address.
Flannel:
The function of Flannel is to allow Docker containers created by different node hosts in the cluster to have unique virtual IP addresses for the entire cluster.
Flannel is a type of Overlay network. It also encapsulates TCP source data packets in another network packet for routing, forwarding and communication. It currently supports UDP, VXLAN (tunnel forwarding), Host-gw (routing entry forwarding, and does not support cloud environments). ) 3 data forwarding methods.
#Flannel How UDP mode works:
After the data is sent from the source container of the Pod on host A, it is forwarded to the flannel0 interface via the docker0/cni0 network interface of the host. The flanneld service listens on the other end of the flannel0 virtual network card.
Flannel maintains a routing table between nodes through the etcd service. The flanneld service of source host A encapsulates the original data content into a UDP message, and delivers it to the flanneld service of destination node host B through the physical network card according to its own routing table. After the data arrives, it is unpacked and then directly enters flannel0 of the destination node. interface, and then forwarded to the docker0/cni0 bridge of the destination host, and finally forwarded by docker0/cni0 to the destination container just like native container communication.
#ETCD’s Flannel provides instructions:
Storage management Flannel allocable IP address segment resources
Monitor the actual address of each Pod in ETCD, and establish and maintain the Pod node routing table in memory
Since UDP mode is forwarded in user mode, packets will be encapsulated through an additional tunnel, so the performance will be worse than VXLAN mode, which is forwarded in kernel mode.
#VXLAN mode:
VXLAN mode is relatively simple to use. flannel will generate a flannel.1 VXLAN network card (VTEP device, responsible for VXLAN encapsulation and decapsulation) on each node.
Operation in VXLAN mode is performed by the kernel. flannel does not forward data, but only dynamically sets ARP table and MAC table entries.
The flannel0 network card in UDP mode is a three-layer forwarding. When using flannel0, a three-layer network is built on the physical network, which belongs to ip in udp; the working mode of VXLAN packetization and unpacking is a two-layer implementation, and the overlay is a data frame, which belongs to mac in udp. .
#Flannel How VXLAN mode works across hosts:
1. After the data frame is sent from the source container of the Pod on host A, it is forwarded to the flannel.1 interface via the docker0/cni0 network interface of the host.
2. After receiving the data frame, flannel.1 adds the VXLAN header and encapsulates it in the UDP message.
3. Host A sends the packet to the physical network card of Host B through the physical network card.
4. The physical network card of host B is forwarded to the flannel.1 interface through the VXLAN default port 4789 for decapsulation.
5. After decapsulation, the kernel sends the data frame to cni0, and finally cni0 sends it to container B bridged to this interface.
//Operate on node01 node
#Upload cni-plugins-linux-amd64-v0.8.6.tgz and flannel.tar to the /opt directory cd /opt/ docker load -i flannel.tar #After executing this command, Docker will load the Docker image from the "flannel.tar" file and add it to the local Docker image library for subsequent use.
mkdir /opt/cni/bin -p tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin
//Operate on master01 node
#Upload the kube-flannel.yml file to the /opt/k8s directory to deploy the CNI network
cd /opt/k8s #kubectl will read and parse the configuration information in the "kube-flannel.yml" file and apply it to the Kubernetes cluster. This YAML file usually contains the configuration of the Flannel network plugin, which is used to set up network communication in the cluster. kubectl apply -f kube-flannel.yml
kubectl get pods -n kube-system
kubectl get nodes
———- Deploying Calico ———-
#k8s networking solution comparison:
●flannel solution
It is necessary to encapsulate the data packets sent to the container on each node, and then use the tunnel to send the encapsulated data packets to the node running the target Pod. The target node is then responsible for removing the encapsulation and sending the de-encapsulated data packet to the target Pod. Data communication performance is greatly affected.
●calico solution
Calico does not use tunnels or NAT to implement forwarding. Instead, it treats the Host as a router in the Internet, uses BGP to synchronize routing, and uses iptables to implement security access policies to complete cross-Host forwarding.
Direct routing is used, which has the lowest performance loss and does not require modification of message data. However, if the network is relatively complex, the routing table will be very complicated, which places higher requirements on operation and maintenance colleagues.
#Calico mainly consists of three parts:
Calico CNI plug-in: Mainly responsible for docking with kubernetes for kubelet calling.
Felix: Responsible for maintaining routing rules, FIB forwarding information database, etc. on the host.
BIRD: Responsible for distributing routing rules, similar to a router.
Confd: configuration management component.
How #Calico works:
Calico maintains communication between each pod through routing tables. Calico’s CNI plug-in will set up a veth pair device for each container, and then connect the other end to the host network space. Since there is no network bridge, the CNI plug-in also needs to configure a veth pair device for each container on the host machine. Routing rules for receiving incoming IP packets.
With such a veth pair device, the IP packets sent by the container will reach the host through the veth pair device, and then the host will send it to the correct gateway according to the next hop address of the routing rules, and then reach the target host, and then arrive at target container.
These routing rules are maintained and configured by Felix, and the routing information is distributed by the Calico BIRD component based on BGP.
calico actually treats all nodes in the cluster as border routers. Together they form a fully interconnected network and exchange routes with each other through BGP. These nodes are called BGP Peers.
Currently, the more commonly used CNI network components are flannel and calico. The function of flannel is relatively simple and does not have complex network policy configuration capabilities. calico is an excellent network management plug-in, but while having complex network configuration capabilities, it often means that its own The configuration is relatively complex, so relatively speaking, flannel is used for smaller and simpler clusters. Considering future expansion, the network may need to add more devices and configure more network policies in the future, so calico is better.
//Operate on master01 node
#Upload the calico.yaml file to the /opt/k8s directory and deploy the CNI network
cd /opt/k8s
vim calico.yaml
#Modify the network (CALICO_IPV4POOL_CIDR) defined in the Pod, which must be the same as the cluster-cidr network segment specified in the previous kube-controller-manager configuration file
- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16" #The default network segment used by Calico is 192.168.0.0/16
#kubectl will read and parse the configuration information in the "calico.yaml" file and apply it to the Kubernetes cluster. This YAML file typically contains the configuration for the Calico network plugin, which is used to set up network communication in the cluster. Calico is a commonly used network plug-in for implementing network policy and network isolation in Kubernetes clusters. kubectl apply -f calico.yaml
#Get detailed information of all Pods in the "kube-system" namespace kubectl get pods -n kube-system
Detailed explanation:
“kubectl get pods -n kube-system” is a command used to get all Pods under the kube-system namespace in a Kubernetes cluster.
- “kubectl” is the Kubernetes command line tool used to interact with Kubernetes clusters.
- “get” is a subcommand of kubectl used to obtain information about Kubernetes resources.
- “Pods” means that the resource type we want to obtain is Pod.
- “-n kube-system” is a parameter that specifies that the namespace of the Pod to be obtained is kube-system.
By executing this command, we can obtain relevant information about all Pods in the kube-system namespace, such as the Pod’s name, status, IP address, node, etc. This information is useful for managing and monitoring Kubernetes clusters.
#When Calico Pod is running, the node will be ready.
#Get node information in the Kubernetes cluster. kubectl get nodes
———- node02 node deployment ———-
//Operate on node01 node
cd /opt/ scp kubelet.sh proxy.sh [email protected]:/opt/ #Copy the local "/opt/cni" directory (including files and subdirectories) to the remote server's "/opt/" directory scp -r /opt/cni [email protected]:/opt/
//Operate on node02 node
#Start kubelet service cd /opt/ chmod +x kubelet.sh #After executing this command, the script file named "kubelet.sh" will be run and "20.0.0.103" will be passed to the script as a parameter. ./kubelet.sh 20.0.0.103
//Operate on master01 node
kubectl get csr
#Request via CSR
kubectl certificate approve node-csr-uPmL1JkOA13jjbyNaG_au0w6pxMUaIxko0m5KAagz0A
#Get the list of Certificate Signing Request (CSR) in the Kubernetes cluster. kubectl get csr
#Load ipvs module (on node02)
for i in $(ls /usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs|grep -o "^[^.]*");do echo $i; / sbin/modinfo -F filename $i >/dev/null 2> & amp;1 & amp; & amp; /sbin/modprobe $i;done
#Use the proxy.sh script to start the proxy service
cd /opt/ chmod +x proxy.sh #Start the agent on 20.0.0.103 ./proxy.sh 20.0.0.103
#View the node status in the cluster
kubectl get nodes
——————————- Deploy CoreDNS —————— ———-
CoreDNS: You can create a domain name and IP resolution for the service resources in the cluster.
//operate on all node nodes
#Upload coredns.tar to the /opt directory
cd /opt docker load -i coredns.tar
//Operate on master01 node
#Upload the coredns.yaml file to the /opt/k8s directory and deploy CoreDNS
cd /opt/k8s kubectl apply -f coredns.yaml
kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-5ffbfd976d-j6shb 1/1 Running 0 32s
#DNS resolution test
kubectl run -it --rm dns-test --image=busybox:1.28.4 sh If you don't see a command prompt, try pressing enter. / # nslookup kubernetes Server: 10.0.0.2 Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local Name: kubernetes Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local Note: If the following error occurs [root@master01 k8s]# kubectl run -it --image=busybox:1.28.4 sh If you don't see a command prompt, try pressing enter. Error attaching, falling back to logs: unable to upgrade connection: Forbidden (user=system:anonymous, verb=create, resource=nodes, subresource=proxy) Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) (pods/log sh) You need to add rbac permissions and directly use kubectl to bind the clusteradmin administrator cluster role and authorize operation permissions. [root@master01 k8s]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created
———- master02 node deployment ———-
#Close firewall
systemctl stop firewalld systemctl disable firewalld iptables -F & amp; & amp; iptables -t nat -F & amp; & amp; iptables -t mangle -F & amp; & amp; iptables -X #Close selinux setenforce 0 sed -i 's/enforcing/disabled/' /etc/selinux/config #Closeswap swapoff -a sed -ri 's/.*swap.*/# & amp;/' /etc/fstab #Set the host name according to the plan hostnamectl set-hostname master02
#Add hosts in master
cat >> /etc/hosts << EOF 20.0.0.101 master01 20.0.0.106 master02 20.0.0.102node01 20.0.0.103 node02 EOF
//Copy the certificate file, configuration files of each master component and service management files from the master01 node to the master02 node
scp -r /opt/etcd/ [email protected]:/opt/ scp -r /opt/kubernetes/ [email protected]:/opt scp -r /root/.kube [email protected]:/root scp /usr/lib/systemd/system/{<!-- -->kube-apiserver,kube-controller-manager,kube-scheduler}.service [email protected]:/usr/lib/systemd/system/
//Modify the IP in the mater02 configuration file kube-apiserver
vim /opt/kubernetes/cfg/kube-apiserver KUBE_APISERVER_OPTS="--logtostderr=true \ --v=4\ --etcd-servers=https://20.0.0.101:2379,https://20.0.0.102:2379,https://20.0.0.103:2379 \ --bind-address=20.0.0.106 \ #Modify --secure-port=6443 \ --advertise-address=20.0.0.106 \ #Modify ...
//Start each service on the master02 node and set it to start automatically at boot
systemctl start kube-apiserver.service systemctl enable kube-apiserver.service systemctl start kube-controller-manager.service systemctl enable kube-controller-manager.service systemctl start kube-scheduler.service systemctl enable kube-scheduler.service
//View node node status in master02
ln -s /opt/kubernetes/bin/* /usr/local/bin/ kubectl get nodes
kubectl get nodes -o wide #-o=wide: Output additional information; for Pod, the Node name where the Pod is located will be output //The node node status found on the master02 node at this time is only the information queried from etcd. At this time, the node node has not actually established a communication connection with the master02 node, so a VIP needs to be used to connect the node node and the master node. related
———————————- Load balancing deployment —————— ————-
//Configure load balancer cluster dual-machine hot backup load balancing (nginx implements load balancing, keepalived implements dual-machine hot backup)
#Operation on lb01 (20.0.0.104), lb02 (20.0.0.105) nodes
//Configure the official online yum source of nginx and configure the yum source of local nginx
cat > /etc/yum.repos.d/nginx.repo << 'EOF' [nginx] name=nginx repo baseurl=http://nginx.org/packages/centos/7/$basearch/ gpgcheck=0 EOF yum install nginx -y
//Modify the nginx configuration file, configure four-layer reverse proxy load balancing, and specify the node IP and 6443 port of the two masters in the k8s cluster.
vim /etc/nginx/nginx.conf
events {<!-- -->
worker_connections 1024;
}
#Add to
stream {<!-- -->
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {<!-- -->
server 20.0.0.101:6443;
server 20.0.0.106:6443;
}
server {<!-- -->
listen 6443;
proxy_pass k8s-apiserver;
}
}
http {<!-- -->
...
//Check configuration file syntax
nginx -t
//Start the nginx service and check that it is listening to port 6443
systemctl start nginx systemctl enable nginx netstat -natp | grep nginx
//Deploy keepalived service
yum install keepalived -y
//Modify keepalived configuration file
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {<!-- -->
#Receive email address
notification_email {<!-- -->
[email protected]
[email protected]
[email protected]
}
# Email sending address
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER #lb01 node is NGINX_MASTER, lb02 node is NGINX_BACKUP
}
#Add a script to be executed periodically
vrrp_script check_nginx {<!-- -->
script "/etc/nginx/check_nginx.sh" #Specify the script path to check nginx survival
}
vrrp_instance VI_1 {<!-- -->
state MASTER #lb01 node is MASTER, lb02 node is BACKUP
interface ens33 #Specify the network card name ens33
virtual_router_id 51 #Specify vrid, the two nodes must be consistent
priority 100 #lb01 node is 100, lb02 node is 90
advert_int 1
authentication {<!-- -->
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {<!-- -->
20.0.0.100/24 #Specify VIP
}
track_script {<!-- -->
check_nginx #Specify the script of vrrp_script configuration
}
}
//Create nginx status check script
vim /etc/nginx/check_nginx.sh
#!/bin/bash
#egrep -cv "grep|$$" is used to filter out the current Shell process ID represented by grep or $$, that is, the current process ID number of the script running
count=$(ps -ef | grep nginx | egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
chmod + x /etc/nginx/check_nginx.sh
//Start the keepalived service (be sure to start the nginx service first, and then start the keepalived service)
systemctl start keepalived systemctl enable keepalived ip a #Check whether VIP is generated
//Modify the bootstrap.kubeconfig and kubelet.kubeconfig configuration files on the node node to VIP
cd /opt/kubernetes/cfg/ vim bootstrap.kubeconfig server: https://20.0.0.100:6443
vim kubelet.kubeconfig server: https://20.0.0.100:6443
vim kube-proxy.kubeconfig server: https://20.0.0.100:6443
//Restart kubelet and kube-proxy services
systemctl restart kubelet.service systemctl restart kube-proxy.service
//Check the connection status of nginx and node and master nodes on lb01
netstat -natp | grep nginx
Operating on master01 node
//Test to create pod
kubectl run nginx --image=nginx
//View Pod status information
kubectl get pods NAME READY STATUS RESTARTS AGE nginx 0/1 ContainerCreating 0 33s #Creating
kubectl get pods NAME READY STATUS RESTARTS AGE nginx 1/1 Running 0 80s #Creation completed, running
kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE nginx 1/1 Running 0 10m 172.17.36.2 192.168.80.15 <none> //READY is 1/1, indicating that there is 1 container in this Pod
//Operate on the node node of the corresponding network segment, you can directly use the browser or curl command to access
curl 172.17.36.2
//At this time, view the nginx log on the master01 node
kubectl logs nginx
——————————- Deploy Dashboard —————— ———-
Introduction to Dashboard
The dashboard is a web-based Kubernetes user interface. You can use the dashboard to deploy containerized applications to a Kubernetes cluster, troubleshoot containerized applications, and manage the cluster itself and its accompanying resources. You can use the dashboard to get an overview of the applications running on the cluster, as well as create or modify individual Kubernetes resources (such as deployments, jobs, daemonsets, etc.). For example, you can use the deployment wizard to extend your deployment, initiate rolling updates, restart Pods, or deploy new applications. The dashboard also provides information about the status of Kubernetes resources in the cluster and any errors that may have occurred.
//Operate on master01 node
#Upload the recommended.yaml file to the /opt/k8s directory
cd /opt/k8s
vim recommended.yaml
#Default Dashboard can only be accessed within the cluster. Modify Service to NodePort type and expose it to the outside:
Kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
nodePort: 30001 #Add
type: NodePort #Add
selector:
k8s-app: kubernetes-dashboard
kubectl apply -f recommended.yaml
#Create service account and bind the default cluster-admin administrator cluster role
kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{<!-- -->print $1}')
#Use the output token to log in to Dashboard
https://NodeIP:30001