1. Problem description
When installing the k8s network plug-in, the following message is prompted, various forbidden permissions are not available
[root@zzyk8s01 scripts]# kubectl apply -f kube-flannel.yml Error from server (Forbidden): error when retrieving current configuration of: Resource: "policy/v1beta1, Resource=podsecuritypolicies", GroupVersionKind: "policy/v1beta1, Kind=PodSecurityPolicy" Name: "psp.flannel.unprivileged", Namespace: "" from server for: "kube-flannel.yml": podsecuritypolicies.policy "psp.flannel.unprivileged" is forbidden: User "system:node:zzyk8s01" cannot get resource "podsecuritypolicies" in API group "policy" at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "flannel", Namespace: "" from server for: "kube-flannel.yml": clusterroles.rbac.authorization.k8s.io "flannel" is forbidden: User "system:node:zzyk8s01" cannot get resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterrolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" Name: "flannel", Namespace: "" from server for: "kube-flannel.yml": clusterrolebindings.rbac.authorization.k8s.io "flannel" is forbidden: User "system:node:zzyk8s01" cannot get resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: "/v1, Resource=serviceaccounts", GroupVersionKind: "/v1, Kind=ServiceAccount" Name: "flannel", Namespace: "kube-system" from server for: "kube-flannel.yml": serviceaccounts "flannel" is forbidden: User "system:node:zzyk8s01" cannot get resource "serviceaccounts" in API group "" in the namespace "kube-system": can only create tokens for individual service accounts Error from server (Forbidden): error when retrieving current configuration of: Resource: "/v1, Resource=configmaps", GroupVersionKind: "/v1, Kind=ConfigMap" Name: "kube-flannel-cfg", Namespace: "kube-system" from server for: "kube-flannel.yml": configmaps "kube-flannel-cfg" is forbidden: User "system:node:zzyk8s01" cannot get resource "configmaps" in API group " " in the namespace "kube-system": no relationship found between node 'zzyk8s01' and this object Error from server (Forbidden): error when retrieving current configuration of: Resource: "apps/v1, Resource=daemonsets", GroupVersionKind: "apps/v1, Kind=DaemonSet" Name: "kube-flannel-ds", Namespace: "kube-system" from server for: "kube-flannel.yml": daemonsets.apps "kube-flannel-ds" is forbidden: User "system:node:zzyk8s01" cannot get resource "daemonsets" in API group "apps" in the namespace "kube-system" [root@zzyk8s01 scripts]#
2. Solution
This may be caused by the config not being updated. /root/.kube/config is copied from /etc/kubernetes/admin.conf. After copying it, you need to execute the update kubeconfig command.
2.1 Update kubeconfig command:
export KUBECONFIG=/root/.kube/config chmod g-r /root/.kube/config
2.2 Execute the command to install flannel again
Prompt various OK
[root@zzyk8s01 scripts]# kubectl apply -f kube-flannel.yml Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21 + , unavailable in v1.25 + podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created
2.3 Check the running status of flannel
Display various Running
[root@zzyk8s01 scripts]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-78fcd69978-vlzpd 1/1 Running 0 2d16h coredns-78fcd69978-vmtdm 1/1 Running 0 2d16h etcd-zzyk8s01 1/1 Running 1 2d16h kube-apiserver-zzyk8s01 1/1 Running 1 2d16h kube-controller-manager-zzyk8s01 1/1 Running 2 2d16h kube-flannel-ds-fpzm9 1/1 Running 0 36s kube-flannel-ds-gkgnz 1/1 Running 0 36s kube-flannel-ds-tmb5s 1/1 Running 0 36s kube-proxy-2gxv9 1/1 Running 0 2d16h kube-proxy-h2zdl 1/1 Running 0 2d16h kube-proxy-v6drm 1/1 Running 1 2d16h kube-scheduler-zzyk8s01 1/1 Running 2 2d16h
2.4 Check flannel network status
As shown below, both cni0 and flannel.1 networks are in UP status
3. Attach the file of kube-flannel.yml
--- apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: psp.flannel.unprivileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default spec: privileged: false volumes: -configMap - secret -emptyDir - hostPath allowedHostPaths: - pathPrefix: "/etc/cni/net.d" - pathPrefix: "/etc/kube-flannel" - pathPrefix: "/run/flannel" readOnlyRootFilesystem: false #Users and groups runAsUser: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny #PrivilegeEscalation allowPrivilegeEscalation: false defaultAllowPrivilegeEscalation: false #Capabilities allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] defaultAddCapabilities: [] requiredDropCapabilities: [] # Host namespaces hostPID: false hostIPC: false hostNetwork: true hostPorts: - min: 0 max: 65535 # SELinux seLinux: # SELinux is unused in CaaS rule: 'RunAsAny' --- kind:ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: flannel rules: - apiGroups: ['extensions'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: ['psp.flannel.unprivileged'] - apiGroups: - "" resources: -pods verbs: - get - apiGroups: - "" resources: -nodes verbs: - list - watch - apiGroups: - "" resources: -nodes/status verbs: -patch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind:ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: kube-flannel-cfg namespace: kube-system labels: tier:node app: flannel data: cni-conf.json: | {<!-- --> "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ {<!-- --> "type": "flannel", "delegate": {<!-- --> "hairpinMode": true, "isDefaultGateway": true } }, {<!-- --> "type": "portmap", "capabilities": {<!-- --> "portMappings": true } } ] } net-conf.json: | {<!-- --> "Network": "10.244.0.0/16", "Backend": {<!-- --> "Type": "vxlan" } } --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds namespace: kube-system labels: tier:node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier:node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect:NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: jmgao1983/flannel #quay.io/coreos/flannel:v0.13.1-rc2 command: -cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: jmgao1983/flannel #quay.io/coreos/flannel:v0.13.1-rc2 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg