k8s install kube-promethues (version 0.7)

k8s install kube-promethues (version 0.7)

1. Check the local k8s version and download the corresponding installation package

kubectl version

As shown in the picture, it is version 1.19

Go to the kube-promethus download address and find out which kube-promethues version is suitable for your k8s version.

Then download the appropriate version for yourself

#You can also directly download the packaged package on the server through the following address. Or copy the address to the browser to download and then upload it to the server.
wget https://github.com/prometheus-operator/kube-prometheus/archive/refs/tags/v0.7.0.tar.gz

This installation was uploaded manually

tar -zxvf kube-prometheus-0.7.0.tar.gz

2. Preparation before installation

1. File classification

When we cd to the corresponding directory, we can see that the initial installation files are very messy.

cd kube-prometheus-0.7.0/manifests/

Create a new directory and categorize the corresponding installation files.

# Create folder
mkdir -p node-exporter alertmanager grafana kube-state-metrics prometheus serviceMonitor adapter

# Move yaml files and classify them into various folders
mv *-serviceMonitor* serviceMonitor/
mv grafana-* grafana/
mv kube-state-metrics-* kube-state-metrics/
mv alertmanager-* alertmanager/
mv node-exporter-* node-exporter/
mv prometheus-adapter* adapter/
mv prometheus-* prometheus

The classified directory tree is as follows

.
├── adapter
│ ├── prometheus-adapter-apiService.yaml
│ ├── prometheus-adapter-clusterRole.yaml
│ ├── prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml
│ ├── prometheus-adapter-clusterRoleBinding.yaml
│ ├── prometheus-adapter-clusterRoleBindingDelegator.yaml
│ ├── prometheus-adapter-clusterRoleServerResources.yaml
│ ├── prometheus-adapter-configMap.yaml
│ ├── prometheus-adapter-deployment.yaml
│ ├── prometheus-adapter-roleBindingAuthReader.yaml
│ ├── prometheus-adapter-service.yaml
│ └── prometheus-adapter-serviceAccount.yaml
├── alertmanager
│ ├── alertmanager-alertmanager.yaml
│ ├── alertmanager-secret.yaml
│ ├── alertmanager-service.yaml
│ └── alertmanager-serviceAccount.yaml
├── grafana
│ ├── grafana-dashboardDatasources.yaml
│ ├── grafana-dashboardDefinitions.yaml
│ ├── grafana-dashboardSources.yaml
│ ├── grafana-deployment.yaml
│ ├── grafana-pvc.yaml
│ ├── grafana-service.yaml
│ └── grafana-serviceAccount.yaml
├── kube-state-metrics
│ ├── kube-state-metrics-clusterRole.yaml
│ ├── kube-state-metrics-clusterRoleBinding.yaml
│ ├── kube-state-metrics-deployment.yaml
│ ├── kube-state-metrics-service.yaml
│ └── kube-state-metrics-serviceAccount.yaml
├── node-exporter
│ ├── node-exporter-clusterRole.yaml
│ ├── node-exporter-clusterRoleBinding.yaml
│ ├── node-exporter-daemonset.yaml
│ ├── node-exporter-service.yaml
│ └── node-exporter-serviceAccount.yaml
├── prometheus
│ ├── prometheus-clusterRole.yaml
│ ├── prometheus-clusterRoleBinding.yaml
│ ├── prometheus-prometheus.yaml
│ ├── prometheus-roleBindingConfig.yaml
│ ├── prometheus-roleBindingSpecificNamespaces.yaml
│ ├── prometheus-roleConfig.yaml
│ ├── prometheus-roleSpecificNamespaces.yaml
│ ├── prometheus-rules.yaml
│ ├── prometheus-service.yaml
│ └── prometheus-serviceAccount.yaml
├── serviceMonitor
│ ├── alertmanager-serviceMonitor.yaml
│ ├── grafana-serviceMonitor.yaml
│ ├── kube-state-metrics-serviceMonitor.yaml
│ ├── node-exporter-serviceMonitor.yaml
│ ├── prometheus-adapter-serviceMonitor.yaml
│ ├── prometheus-operator-serviceMonitor.yaml
│ ├── prometheus-serviceMonitor.yaml
│ ├── prometheus-serviceMonitorApiserver.yaml
│ ├── prometheus-serviceMonitorCoreDNS.yaml
│ ├── prometheus-serviceMonitorKubeControllerManager.yaml
│ ├── prometheus-serviceMonitorKubeScheduler.yaml
│ └── prometheus-serviceMonitorKubelet.yaml
└── setup
    ├── 0namespace-namespace.yaml
    ├── prometheus-operator-0alertmanagerConfigCustomResourceDefinition.yaml
    ├── prometheus-operator-0alertmanagerCustomResourceDefinition.yaml
    ├── prometheus-operator-0podmonitorCustomResourceDefinition.yaml
    ├── prometheus-operator-0probeCustomResourceDefinition.yaml
    ├── prometheus-operator-0prometheusCustomResourceDefinition.yaml
    ├── prometheus-operator-0prometheusruleCustomResourceDefinition.yaml
    ├── prometheus-operator-0servicemonitorCustomResourceDefinition.yaml
    ├── prometheus-operator-0thanosrulerCustomResourceDefinition.yaml
    ├── prometheus-operator-clusterRole.yaml
    ├── prometheus-operator-clusterRoleBinding.yaml
    ├── prometheus-operator-deployment.yaml
    ├── prometheus-operator-service.yaml
    └── prometheus-operator-serviceAccount.yaml

8 directories, 68 files

2. Check whether the K8s cluster has NFS persistent storage installed. If not, you need to install and configure it

kubectl get sc

This screenshot shows it has been installed. The following is the installation and deployment method of NFS

1).Install NFS service

Ubuntu:

sudo apt update
sudo apt install nfs-kernel-server

Centos:

yum update
yum -y install nfs-utils

# Create or use an existing folder as an nfs file storage point
mkdir -p /home/data/nfs/share
vi /etc/exports

Write the following content

/home/data/nfs/share *(rw,no_root_squash,sync,no_subtree_check)

# Validate the configuration and check whether it takes effect
exportfs -r
exportfs

# Start rpcbind and nfs services
#Centos
systemctl restart rpcbind & amp; & amp; systemctl enable rpcbind
systemctl restart nfs & amp; & amp; systemctl enable nfs
#Ubuntu
systemctl restart rpcbind & amp; & amp; systemctl enable rpcbind
systemctl start nfs-kernel-server & amp; & amp; systemctl enable nfs-kernel-server

# Check the registration status of RPC service
rpcinfo -p localhost

# showmount test
showmount -e localhost

If there are no problems with the above, it means the installation is successful.

2).k8s registration nfs service

Create a new storageclass-nfs.yaml file and paste the following content:

## Created a storage class
apiVersion: storage.k8s.io/v1
kind: StorageClass #The resource name of the storage class
metadata:
  name: nfs-storage #The name of the storage class, customized
  annotations:
    storageclass.kubernetes.io/is-default-class: "true" #Annotation, whether it is the default storage. Note: KubeSphere requires a default storage by default, so the annotation here should be set to the "default" storage system, indicating Is "true", representing the default.
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner #The name of the storage allocator, customized
parameters:
  archiveOnDelete: "true" ## When deleting a pv, whether the content of the pv needs to be backed up

---
apiVersion: apps/v1
Kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace:default
spec:
  replicas: 1 #Only run one replica application
  strategy: #Describes how to replace existing PODs with new PODs
    type: Recreate #Recreate means recreating the Pod
  selector: #Select backend Pod
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner #Create account
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2 #Use the image of NFS storage allocator
          # resources:
          # limits:
          # cpu: 10m
          # requests:
          # cpu: 10m
          volumeMounts:
            - name: nfs-client-root #Define a storage volume,
              mountPath: /persistentvolumes #Indicates the path inside the mounting container
          env:
            - name: PROVISIONER_NAME #Define the name of the storage allocator
              value: k8s-sigs.io/nfs-subdir-external-provisioner #Need to keep the name consistent with the one defined above
            - name: NFS_SERVER #Specify the address of the NFS server. You need to change it to the IP address of your NFS server.
              value: 192.168.0.0 ## Specify your own nfs server address
            - name: NFS_PATH
              value: /home/data/nfs/share ## Directory shared by nfs server #Specify the directory shared by NFS server
      volumes:
        - name: nfs-client-root #The name of the storage volume, consistent with the previous definition
          nfs:
            server: 192.168.0.0 #The address of the NFS server is consistent with the above. It needs to be changed to your IP address.
            path: /home/data/nfs/share #NFS shared storage directory, consistent with the above
---
apiVersion: v1
kind: ServiceAccount #Create a SA account
metadata:
  name: nfs-client-provisioner #Same as the SA account above
  # replace with namespace where provisioner is deployed
  namespace:default
---
#The following are ClusterRole, ClusterRoleBinding, Role, and RoleBinding are permission binding configurations and will not be explained. Just copy it directly.
kind:ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace:default
roleRef:
  kind:ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace:default
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace:default
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace:default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

The only things that need to be modified are the server address and shared directory.

Create StorageClass

kubectl apply -f storageclass-nfs.yaml

# Check if it exists
kubectl get sc

3. Modify Prometheus persistence

vi prometheus/prometheus-prometheus.yaml

Add at the end of the file:

...
  serviceMonitorSelector: {<!-- -->}
  version: v2.11.0
  retention: 3d
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: nfs-storage
        resources:
          requests:
            Storage: 5Gi

4. Modify grafana persistence configuration

#Add garfana’s PVC configuration file
vi grafana/grafana-pvc.yaml

The complete content is as follows:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: grafana
  namespace: monitoring #---Specify namespace as monitoring
spec:
  storageClassName: nfs-storage #---Specify StorageClass
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      Storage: 5Gi

Then modify the grafana-deployment.yaml file to set the persistence configuration, and modify the image version of Garfana (some templates do not support Grafana below 7.5), and apply the above PVC

vi grafana/grafana-deployment.yaml

The modifications are as follows:

 serviceAccountName: grafana
      volumes:
      - name: grafana-storage # Add persistence configuration
        persistentVolumeClaim:
          claimName: grafana # Set to the name of the created PVC
# - emptyDir: {} # Comment old comments
# name: grafana-storage
      - name: grafana-datasources
        secret:
          secretName: grafana-datasources

Previous image version

after edited

5. Modify the Service port settings of promethus and Grafana

Modify Prometheus Service

vi prometheus/prometheus-service.yaml

Modify to the following content:

apiVersion: v1
Kind: Service
metadata:
  labels:
    prometheus:k8s
  name: prometheus-k8s
  namespace: monitoring
spec:
  type: NodePort
  ports:
  - name: web
    port: 9090
    targetPort: web
    nodePort: 32101
  selector:
    app: prometheus
    prometheus:k8s
  sessionAffinity: ClientIP

Modify Grafana Service

vi grafana/grafana-service.yaml

Modify to the following content:

apiVersion: v1
Kind: Service
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  type: NodePort
  ports:
  - name: http
    port: 3000
    targetPort: http
    nodePort: 32102
  selector:
    app: grafana

3. Install Prometheus

1. Install promethues-operator

First make sure it is in the manifests directory

Start installing the Operator:

kubectl apply -f setup/

Check the Pod and wait until all pods are ready before proceeding to the next step:

kubectl get pods -n monitoring

2. Install all other components

#Execute in sequence
kubectl apply -f adapter/
kubectl apply -f alertmanager/
kubectl apply -f node-exporter/
kubectl apply -f kube-state-metrics/
kubectl apply -f grafana/
kubectl apply -f prometheus/
kubectl apply -f serviceMonitor/

Then check whether the pod is created successfully and wait for all pods to be in the Running state

kubectl get pods -n monitoring

3. Verify whether the installation is successful

If you know the cluster node address, you can access Prometheus directly at ip:32101. If you don’t know, you can access the Rancher management interface and select monitoring for the namespace. Find prometheus-k8s and grafana in Services and click on the target port to access them.

Just test a function on the Prometheus interface to see if it can be used normally.

Then log in to Grafana

The default username and password are admin/admin. You will be prompted to change the password when logging in for the first time. After entering Grafana, import the template test. The recommended template IDs are 12884 and 13105

Rendering: