Table of Contents
emptyDir storage volume
hostPath storage volume
nfs shared storage volume
PVC and PV
PV creation and destruction process
NFS uses PV and PVC
Build StorageClass + NFS to realize dynamic PV creation of NFS
The life cycle of files on the container disk is short-lived, which causes some problems when running important applications in the container. First, when a container crashes, the kubelet will restart it, but the files in the container will be lost – the container is restarted in a clean state (the original state of the image). Secondly, when multiple containers are running simultaneously in a Pod, files usually need to be shared between these containers. The Volume abstraction in Kubernetes solves these problems very well. Containers in the Pod share the Volume through the Pause container.
emptyDir storage volume
When a Pod is assigned to a node, the emptyDir volume is first created and exists as long as the Pod is running on that node. As the name of the volume states, it is initially empty. Containers in a Pod can read and write the same files in the emptyDir volume, although the volume can be mounted to the same or different paths in each container. When a Pod is removed from a node for any reason, the data in emptyDir is permanently deleted.
mkdir /opt/volumes
cd /opt/volumes
vim pod-emptydir.yaml
apiVersion: v1
Kind: Pod
metadata:
name: pod-emptydir
namespace:default
labels:
app: myapp
tier: frontend
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
#Define container mounting content
volumeMounts:
#The storage volume name used. If the value of the volume field name below is the same, it means that the storage volume of the volume is used.
- name: html
#Which directory to mount to in the container
mountPath: /usr/share/nginx/html/
- name: busybox
image: busybox:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- name: html
#Define the mount storage name and mount path within the container
mountPath: /data/
command: ['/bin/sh','-c','while true;do echo $(date) >> /data/index.html;sleep 2;done']
#Define storage volume
volumes:
#Define storage volume name
- name: html
#Define storage volume type
emptyDir: {}
\t
\t
kubectl apply -f pod-emptydir.yaml
kubectl get pods -o wide
//Two containers are defined above, one of which is to input the date into index.html, and then verify whether the date can be obtained by accessing nginx’s html. To verify that the emptyDir mounted between the two containers is shared.
curl 10.244.2.3
hostPath storage volume
A hostPath volume mounts a file or directory in the node’s file system into the cluster.
hostPath can achieve persistent storage, but it will also cause data loss when the node node fails.
//Create a mounting directory on the node01 node
mkdir -p /data/pod/volume1 echo 'node01.my.com' > /data/pod/volume1/index.html
//Create a mounting directory on the node02 node
mkdir -p /data/pod/volume1 echo 'node02.my.com' > /data/pod/volume1/index.html
//Create Pod resources
vim pod-hostpath.yaml
apiVersion: v1
Kind: Pod
metadata:
name: pod-hostpath
namespace:default
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
#Define container mounting content
volumeMounts:
#The storage volume name used. If the value of the volume field name below is the same, it means that the storage volume of the volume is used.
- name: html
#Which directory to mount to in the container
mountPath: /usr/share/nginx/html
#Read-write mounting mode, the default is read-write mode false
readOnly: false
The #volumes field defines the host or distributed file system storage volume associated with the paues container
volumes:
#Storage volume name
- name: html
#Path, storage path for the host machine
hostPath:
#The path to the directory on the host machine
path: /data/pod/volume1
#Define the type, which means that if the host does not have this directory, it will be automatically created.
type: DirectoryOrCreate
kubectl apply -f pod-hostpath.yaml
//Access test
kubectl get pods -o wide
curl 10.244.1.4
//Delete the pod, rebuild it again, and verify whether the original content can still be accessed
kubectl delete -f pod-hostpath.yaml kubectl apply -f pod-hostpath.yaml
kubectl get pods -o wide
curl 10.244.1.5
nfs shared storage volume
//Install nfs on the stor01 node and configure the nfs service
hostnamectl set-hostname stor01 mkdir /data/volumes -p chmod 777 /data/volumes rpm -q rpcbind nfs-utils
vim /etc/exports /data/volumes 20.0.0.0/24(rw,no_root_squash)
systemctl start rpcbind systemctl start nfs systemctl enable rpcbind nfs kubectl apply -f pod-nfs-vol.yaml showmount -e
on node01 and node02
showmount -e 20.0.0.104
Operate on the master node
vim pod-nfs-vol.yaml
apiVersion: v1
Kind: Pod
metadata:
name: pod-nfs
namespace:default
labels:
app:myapp
spec:
containers:
- name: myapp
image: soscscs/myapp:v1
volumeMounts:
- name: xiaoma
mountPath: /usr/share/nginx/html
readOnly: false
nodeSelector:
kubernetes.io/hostname: node02
volumes:
- name: xiaoma
nfs:
path: /data/volumes
server: 20.0.0.104
kubectl apply -f pod-nfs-vol.yaml
kubectl get pods -o wide
//Create index.html on the nfs server
cd /data/volumes vim index.html <h1>nfs stor01</h1>
//master node operation
Test: curl 10.244.2.4
kubectl delete -f pod-nfs-vol.yaml #Delete nfs-related pods and re-create them to obtain persistent storage of data
kubectl apply -f pod-nfs-vol.yaml
PVC and PV
The full name of PV is Persistent Volume, which is a persistent storage volume. It is used to describe or define a storage volume, which is usually defined by operation and maintenance engineers.
The full name of PVC is Persistent Volume Claim, which is a request for persistent storage. It is used to describe what kind of PV storage you want to use or what conditions you want to meet.
The usage logic of PVC: Define a storage volume in the Pod (the storage volume type is PVC), specify the size directly when defining, the PVC must establish a relationship with the corresponding PV, the PVC will apply to the PV according to the configuration definition, and the PV is Created from storage space. PV and PVC are storage resources abstracted by Kubernetes.
The PV and PVC modes introduced above require operation and maintenance personnel to create PVs first, and then developers define PVCs for one-to-one bonding. However, if there are thousands of PVC requests, thousands of PVs need to be created. Maintenance costs are very high for operation and maintenance personnel. Kubernetes provides a mechanism to automatically create PVs called StorageClass, which is used to create PV templates.
Creating a StorageClass requires defining the attributes of the PV, such as storage type, size, etc.; in addition, creating such a PV requires the use of storage plug-ins, such as Ceph, etc. With these two pieces of information, Kubernetes can find the corresponding StorageClass based on the PVC submitted by the user. Then Kubernetes will call the storage plug-in declared by the StorageClass to automatically create the required PV and bind it.
PV is a resource in the cluster. PVC is a request for these resources and an index check of the resources.
The interaction between PV and PVC follows this life cycle:
Provisioning ---> Binding ---> Using ---> Releasing ---> Recycling ●Provisioning, that is, the creation of PV, you can create PV directly (static method), or you can use StorageClass to create it dynamically ●Binding, assigning PV to PVC ●Using, Pod uses the Volume through PVC, and can prevent deletion of the PVC in use through admission control StorageProtection (1.9 and previous versions are PVCProtection) ●Releasing, Pod releases Volume and deletes PVC ●Reclaiming, recycling PV, you can keep the PV for next use, or you can delete it directly from the cloud storage According to these 5 stages, there are 4 states of PV: ●Available: Indicates available status and has not been bound by any PVC. ●Bound: indicates that the PV has been bound to the PVC ●Released: Indicates that the PVC has been deleted, but the resources have not been reclaimed by the cluster. ●Failed: Indicates that the automatic recycling of the PV failed
PV creation and destruction process
The specific process of a PV from creation to destruction is as follows:
1. After a PV is created, its status will change to Available, waiting to be bound by the PVC.
2. Once bound by PVC, the status of PV will change to Bound, and it can be used by Pods with corresponding PVC defined.
3. After the Pod is used, the PV will be released, and the status of the PV will change to Released.
4. The PV that becomes Released will be recycled according to the defined recycling strategy. There are three recycling strategies, Retain, Delete and Recycle. Retain means to retain the scene. The K8S cluster does nothing and waits for the user to manually process the data in the PV. After the processing is completed, the PV is manually deleted. Delete policy, K8S will automatically delete the PV and the data in it. In Recycle mode, K8S will delete the data in the PV, and then change the status of the PV to Available, which can then be bound and used by new PVCs.
Set local mapping on node01 and node02
cat>> /etc/hosts<<EOF 20.0.0.104 stor01 EOF
kubectl explain pv #View how pv is defined
FIELDS:
apiVersion: v1
kind: PersistentVolume
metadata: #Since PV is a cluster-level resource, that is, PV can be used across namespaces, so there is no need to configure the namespace in the PV's metadata.
name:
spec
\t
kubectl explain pv.spec #View the specifications defined by pv
spec:
nfs: (define storage type)
path: (define the mount volume path)
server: (define server name)
accessModes: (define the access model, there are the following three access models, which exist in the form of a list, which means multiple access modes can be defined)
- ReadWriteOnce #(RWO) storage is readable and writable, but only supports being mounted by a single Pod
- ReadOnlyMany #(ROX) storage can be mounted by multiple Pods in a read-only manner
- ReadWriteMany # (RWX) storage can be shared by multiple Pods in a read and write manner Note: Official website
#nfs supports all three; iSCSI does not support ReadWriteMany (iSCSI is a network storage technology that runs the SCSI protocol on an IP network); HostPath does not support ReadOnlyMany and ReadWriteMany.
capacity: (defines storage capacity, generally used to set storage space)
storage: 2Gi (specify size)
storageClassName: (custom storage class name, this configuration is used to bind PVC and PV with the same category)
persistentVolumeReclaimPolicy: Retain #Recycling policy (Retain/Delete/Recycle)
#Retain: When the PVC bound to it is deleted, the PV is marked as released (the PVC is unbound from the PV but the recycling strategy has not been implemented) and the previous data is still saved on the PV, but the PV cannot be You need to manually process the data and delete the PV.
#Delete: Delete the backend storage resources connected to the PV (only supported by AWS EBS, GCE PD, Azure Disk and Cinder)
#Recycle (recycling): delete data, the effect is equivalent to executing rm -rf /thevolume/* (only supported by NFS and HostPath)
kubectl explain pvc #View how PVC is defined
KIND: PersistentVolumeClaim
VERSION: v1
FIELDS:
apiVersion <string>
kind <string>
metadata <Object>
spec<Object>
#The spec key fields in PV and PVC must match, such as storage (storage) size, access mode (accessModes), and storage class name (storageClassName)
kubectl explain pvc.spec
spec:
accessModes: (define the access mode, which must be a subset of the PV’s access mode)
resources:
requests:
storage: (define the size of the requested resource)
storageClassName: (defines the storage class name, this configuration is used to bind PVC and PV with the same category)
NFS uses PV and PVC
1. Configure nfs storage
Created on sto01
mkdir v{1,2,3,4,5}
vim /etc/exports
/data/volumes/v1 20.0.10.0/24(rw,no_root_squash)
/data/volumes/v2 20.0.0.0/24(rw,no_root_squash)
/data/volumes/v3 20.0.0.0/24(rw,no_root_squash)
/data/volumes/v4 20.0.0.0/24(rw,no_root_squash)
/data/volumes/v5 20.0.0.0/24(rw,no_root_squash)
exportfs -arv
showmount -e
Official documentation: https://kubernetes.io/zh-cn/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#create-a-persistentvolume
2. Define PV (on master)
//Define 5 PVs here, and define the mounting path and access mode, as well as the size of the PV division.
vim pv-demo.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv001
labels:
name: pv001
spec:
nfs:
path: /data/volumes/v1
server: stor01
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
Storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv002
labels:
name: pv002
spec:
nfs:
path: /data/volumes/v2
server: stor01
accessModes: ["ReadWriteOnce"]
capacity:
Storage: 2Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv003
labels:
name: pv003
spec:
nfs:
path: /data/volumes/v3
server: stor01
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
Storage: 2Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv004
labels:
name: pv004
spec:
nfs:
path: /data/volumes/v4
server: stor01
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
storage: 4Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv005
labels:
name: pv005
spec:
nfs:
path: /data/volumes/v5
server: stor01
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
Storage: 5Gi
kubectl apply -f pv-demo.yaml
#View pv kubectl get pv
3. Define PVC
//The access mode of pvc is defined here as multi-channel read and write. This access mode must be among the access modes defined by pv previously. Define the size of the PVC application to be 2Gi. At this time, the PVC will automatically match the multi-channel read and write PV with a size of 2Gi. The status of the PVC successfully obtained by matching is Bound.
vim pod-vol-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name:mypvc
namespace:default
spec:
accessModes: ["ReadWriteMany"]
resources:
requests:
Storage: 2Gi
---
apiVersion: v1
Kind: Pod
metadata:
name: pod-vol-pvc
namespace:default
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
volumeMounts:
- name: html
mountPath: /usr/share/nginx/html
volumes:
- name: html
persistentVolumeClaim:
claimName: mypvc
kubectl apply -f pod-vol-pvc.yaml
kubectl get pv
kubectl get pvc
4. Test access
//Create index.html on the storage server, write data, and view the corresponding page by accessing the Pod.
cd /data/volumes/v3/ echo "welcome to use pv3" > index.html
kubectl get pods -o wide
curl 10.244.1.6
Build StorageClass + NFS to realize dynamic PV creation of NFS
The dynamic PV creation supported by Kubernetes itself does not include NFS, so you need to use an external storage volume plug-in to allocate PVs. For details, see:
https://kubernetes.io/zh/docs/concepts/storage/storage-classes/
The volume plug-in is called Provisioner (storage allocator), and NFS uses nfs-client. This external volume plug-in will automatically create PVs using the configured NFS server.
Provisioner: used to specify the type of Volume plug-in, including built-in plug-ins (such as kubernetes.io/aws-ebs) and external plug-ins (such as ceph.com/cephfs provided by external-storage).
1. Install nfs on the stor01 node and configure the nfs service
mkdir /opt/k8s chmod 777 /opt/k8s/
vim /etc/exports /opt/k8s 20.0.0.0/24(rw,no_root_squash,sync)
systemctl restart nfs
2. Create a Service Account to manage the permissions of NFS Provisioner to run in the k8s cluster, and set nfs-client rules for PV, PVC, StorageClass, etc.
vim nfs-client-rbac.yaml
#Create a Service Account to manage the permissions of NFS Provisioner running in the k8s cluster
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
---
#Create cluster role
apiVersion: rbac.authorization.k8s.io/v1
kind:ClusterRole
metadata:
name: nfs-client-provisioner-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
---
#Cluster role binding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: nfs-client-provisioner-clusterrolebinding
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace:default
roleRef:
kind:ClusterRole
name: nfs-client-provisioner-clusterrole
apiGroup: rbac.authorization.k8s.io
kubectl apply -f nfs-client-rbac.yaml
3. Use Deployment to create NFS Provisioner
NFS Provisione (ie nfs-client) has two functions: one is to create a mount point (volume) under the NFS shared directory, and the other is to associate the PV with the NFS mount point.
#Since selfLink is enabled in version 1.20, an error will be reported when k8s 1.20+ version dynamically generates pv through nfs provisioner. The solution is as follows:
vim /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
containers:
- command:
- kube-apiserver
- --feature-gates=RemoveSelfLink=false #Add this line
- --advertise-address=192.168.80.20
...
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml kubectl delete pods kube-apiserver -n kube-system kubectl get pods -n kube-system | grep apiserver
#Create NFS Provisioner
First, pass nfs-client-provisioner.tar to node01 and node02 to speed up image pulling.
docker load -i nfs-client-provisioner.tar scp nfs-client-provisioner.tar @20.0.0.103:/opt/
docker images View images
vim nfs-client-provisioner.yaml
Kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner #Specify Service Account account
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: nfs-storage #Configure the provisioner Name and ensure that the name is consistent with the provisioner name in the StorageClass resource
- name: NFS_SERVER
value: stor01 #Configure the bound nfs server
- name: NFS_PATH
value: /opt/k8s #Configure the bound nfs server directory
volumes: #Declare nfs data volume
- name: nfs-client-root
nfs:
server: stor01
path: /opt/k8s
\t
\t
kubectl apply -f nfs-client-provisioner.yaml
kubectl get pod
4. Create StorageClass, which is responsible for establishing PVC and calling NFS provisioner to perform scheduled work, and associate PV with PVC.
vim nfs-client-storageclass.yaml
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-client-storageclass provisioner: nfs-storage #The name here should be consistent with the environment variable PROVISIONER_NAME in the provisioner configuration file parameters: archiveOnDelete: "false" #false means that the data will not be archived when the PVC is deleted, that is, the data will be deleted
kubectl apply -f nfs-client-storageclass.yaml
kubectl get storageclass
5. Create PVC and Pod tests
vim test-pvc-pod.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-nfs-pvc
spec:
accessModes:
- ReadWriteMany
storageClassName: nfs-client-storageclass #Associate StorageClass object
resources:
requests:
Storage: 1Gi
---
apiVersion: v1
Kind: Pod
metadata:
name: test-storageclass-pod
spec:
containers:
- name: busybox
image: busybox:latest
imagePullPolicy: IfNotPresent
command:
- "/bin/sh"
- "-c"
args:
- "sleep 3600"
volumeMounts:
- name: nfs-pvc
mountPath: /mnt
restartPolicy: Never
volumes:
- name: nfs-pvc
persistentVolumeClaim:
claimName: test-nfs-pvc #Consistent with the PVC name
\t
\t
kubectl apply -f test-pvc-pod.yaml
//PVC automatically applies for space through StorageClass
kubectl get pvc
//Check whether the corresponding directory is generated on the NFS server. The automatically created PV will be placed on the NFS server in the directory format of n a m e s p a c e ? {namespace}- namespace? {pvcName}-${pvName}
ls /opt/k8s/
//Enter Pod and write a file under the mounting directory /mnt, and then check whether the file exists on the NFS server
kubectl exec -it test-storageclass-pod sh / # cd /mnt/ /mnt # echo 'ceshiyixia' > test.txt
// Found to exist on the NFS server, indicating successful verification
cat /opt/k8s/default-test-nfs-pvc-pvc-8885e05c-bc65-4930-a2dc-dc3b42c38111/test.txt
The knowledge points of the article match the official knowledge archives, and you can further learn relevant knowledge. Cloud native entry-level skills treeContainer orchestration (production environment k8s)kubelet, kubectl, kubeadm three-piece set 17034 people Currently studying the system