Rolling release, graceful downtime, elastic scaling, application monitoring, configuration separation in Spring Boot + K8S

96edba27b69ccb07aad7bf7834e81fc0.png

Source: blog.csdn.net/qq_14999375/article/details/123309636

K8s + SpringBoot achieves zero downtime release: health check + rolling update + graceful shutdown + elastic scaling + Prometheus monitoring + configuration separation (mirror reuse)

Configuration

Health check

  • Health check type: readiness probe (readiness) + survival probe (liveness)

  • Probe type: exec (enter the container to execute the script), tcpSocket (detect port), httpGet (call interface)

Business level

The project depends on pom.xml

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Define access ports, paths and permissions application.yaml

management:
  server:
    port: 50000 # enable independent operation and maintenance port
  endpoint: # open the health endpoint
    health:
      probes:
        enabled: true
  endpoints:
    web:
      exposure:
        base-path: /actuator # Specify the context path and enable the corresponding endpoint
        include: health

Two interfaces /actuator/health/readiness and /actuator/health/liveness will be exposed, and the access methods are as follows:

http://127.0.0.1:50000/actuator/health/readiness
http://127.0.0.1:50000/actuator/health/liveness
Operation and maintenance level

k8s deployment template deployment.yaml

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: {APP_NAME}
        image: {IMAGE_URL}
        imagePullPolicy: Always
        ports:
        - containerPort: {APP_PORT}
        - name: management-port
          containerPort: 50000 # application management port
        readinessProbe: # readiness probe
          httpGet:
            path: /actuator/health/readiness
            port: management-port
          initialDelaySeconds: 30 # delay loading time
          periodSeconds: 10 # retry interval
          timeoutSeconds: 1 # timeout setting
          successThreshold: 1 # health threshold
          failureThreshold: 6 # unhealthy threshold
        livenessProbe: # liveness probe
          httpGet:
            path: /actuator/health/liveness
            port: management-port
          initialDelaySeconds: 30 # delay loading time
          periodSeconds: 10 # retry interval
          timeoutSeconds: 1 # timeout setting
          successThreshold: 1 # health threshold
          failureThreshold: 6 # unhealthy threshold

Rolling update

The rolling update strategy of k8s resource scheduling, in order to achieve zero downtime release, needs to support health check

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {APP_NAME}
  labels:
    app: {APP_NAME}
spec:
  selector:
    matchLabels:
      app: {APP_NAME}
  replicas: {REPLICAS} # Number of Pod replicas
  strategy:
    type: RollingUpdate # rolling update strategy
    rollingUpdate:
      maxSurge: 1 # The maximum number of replicas that can be set during the upgrade process
      maxUnavailable: 1 # The maximum number of PODs that cannot provide services during the upgrade process
Graceful shutdown

In K8s, before we implement rolling upgrades, we must implement graceful shutdown at the application level. Otherwise, the rolling upgrade will still affect the business. Make the application close the thread, release the connection resource and then stop the service

Business level

The project depends on pom.xml

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Define access ports, paths and permissions application.yaml

spring:
  application:
    name: <xxx>
  profiles:
    active: @profileActive@
  lifecycle:
    timeout-per-shutdown-phase: 30s # The timeout period of the shutdown process is set to 30s, and if it exceeds 30s, it will stop directly

server:
  port: 8080
  shutdown: graceful # The default is IMMEDIATE, which means immediate shutdown; GRACEFUL means graceful shutdown

management:
  server:
    port: 50000 # enable independent operation and maintenance port
  endpoint: # Turn on the shutdown and health endpoints
    shutdown:
      enabled: true
    health:
      probes:
        enabled: true
  endpoints:
    web:
      exposure:
        base-path: /actuator # Specify the context path and enable the corresponding endpoint
        include: health, shutdown

The /actuator/shutdown interface will be exposed, and the calling method is as follows:

curl -X POST 127.0.0.1:50000/actuator/shutdown
Operation and maintenance level

Make sure that the dockerfile template integrates the curl tool, otherwise the curl command cannot be used

FROM openjdk:8-jdk-alpine
#build parameters
ARG JAR_FILE
ARG WORK_PATH="/app"
ARG EXPOSE_PORT=8080

#Environment variables
ENV JAVA_OPTS=""\
    JAR_FILE=${JAR_FILE}

# set time zone
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime & amp; & amp; echo 'Asia/Shanghai' >/etc/timezone
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories \
     & amp; & amp; apk add --no-cache curl
#Copy the jar package in the maven directory to docker and name it for_docker.jar
COPY target/$JAR_FILE $WORK_PATH/


#Set working directory
WORKDIR $WORK_PATH


# Specify the port for external interaction
EXPOSE $EXPOSE_PORT
# Configure the container to make it executable
ENTRYPOINT exec java $JAVA_OPTS -jar $JAR_FILE

k8s deployment template deployment.yaml

Note: It has been verified that the java project can omit the configuration of the end callback hook

In addition, if you need to use the callback hook, you need to ensure that the curl tool is included in the image, and you need to pay attention to the fact that the application management port (50000) cannot be exposed to the public network

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: {APP_NAME}
        image: {IMAGE_URL}
        imagePullPolicy: Always
        ports:
        - containerPort: {APP_PORT}
        - containerPort: 50000
        lifecycle:
          preStop: # end callback hook
            exec:
              command: ["curl", "-XPOST", "127.0.0.1:50000/actuator/shutdown"]

Elastic scaling

After setting resource limits for the pods, create the HPA

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {APP_NAME}
  labels:
    app: {APP_NAME}
spec:
  template:
    spec:
      containers:
      - name: {APP_NAME}
        image: {IMAGE_URL}
        imagePullPolicy: Always
        resources: # container resource management
          limits: # resource limits (monitor usage)
            cpu: 0.5
            memory: 1Gi
          requests: # Minimum available resources (flexible scheduling)
            cpu: 0.15
            memory: 300Mi
---
kind: HorizontalPodAutoscaler # elastic scaling controller
apiVersion: autoscaling/v2beta2
metadata:
  name: {APP_NAME}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {APP_NAME}
  minReplicas: {REPLICAS} # scaling range
  maxReplicas: 6
  metrics:
    -type: Resource
      resource:
        name: cpu # Specify the resource index
        target:
          type: Utilization
          averageUtilization: 50

Prometheus integration

Business level

The project depends on pom.xml


<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

    io.micrometer
    micrometer-registry-prometheus

Define access ports, paths and permissions application.yaml

management:
  server:
    port: 50000 # enable independent operation and maintenance port
  metrics:
    tags:
      application: ${spring.application.name}
  endpoints:
    web:
      exposure:
        base-path: /actuator # Specify the context path and enable the corresponding endpoint
        include: metrics,prometheus

The /actuator/metric and /actuator/prometheus interfaces will be exposed, and the access methods are as follows:

http://127.0.0.1:50000/actuator/metric
http://127.0.0.1:50000/actuator/prometheus
Operation and maintenance level

deployment.yaml

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        prometheus:io/port: "50000"
        prometheus.io/path: /actuator/prometheus # assign value in the pipeline
        prometheus.io/scrape: "true" # Pod-based service discovery

Configuration separation

Solution: Mount the external configuration file through configmap, and specify the activation environment to run

Role: Separation of configurations to avoid leakage of sensitive information; multiplexing of images to improve delivery efficiency

Generate configmap from file

# Generate yaml file by dry-run
kubectl create cm -n <namespace> <APP_NAME> --from-file=application-test.yaml --dry-run=1 -oyaml > configmap.yaml

# renew
kubectl apply -f configmap.yaml

Mount configmap and specify the activation environment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {APP_NAME}
  labels:
    app: {APP_NAME}
spec:
  template:
    spec:
      containers:
      - name: {APP_NAME}
        image: {IMAGE_URL}
        imagePullPolicy: Always
        env:
          - name: SPRING_PROFILES_ACTIVE # Specify the activation environment
            value: test
        volumeMounts: # mount configmap
        -name: conf
          mountPath: "/app/config" # Consistent with the working directory in Dockerfile
          readOnly: true
      volumes:
      -name: conf
        configMap:
          name: {APP_NAME}

Summary configuration

Business level

The project depends on pom.xml


<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

    io.micrometer
    micrometer-registry-prometheus

Define access ports, paths and permissions application.yaml

spring:
  application:
    name: project-sample
  profiles:
    active: @profileActive@
  lifecycle:
    timeout-per-shutdown-phase: 30s # The timeout period of the shutdown process is set to 30s, and if it exceeds 30s, it will stop directly

server:
  port: 8080
  shutdown: graceful # The default is IMMEDIATE, which means immediate shutdown; GRACEFUL means graceful shutdown

management:
  server:
    port: 50000 # enable independent operation and maintenance port
  metrics:
    tags:
      application: ${spring.application.name}
  endpoint: # Turn on the shutdown and health endpoints
    shutdown:
      enabled: true
    health:
      probes:
        enabled: true
  endpoints:
    web:
      exposure:
        base-path: /actuator # Specify the context path and enable the corresponding endpoint
        include: health, shutdown, metrics, prometheus

Operation and maintenance level

Make sure that the dockerfile template integrates the curl tool, otherwise the curl command cannot be used

FROM openjdk:8-jdk-alpine
#build parameters
ARG JAR_FILE
ARG WORK_PATH="/app"
ARG EXPOSE_PORT=8080

#Environment variables
ENV JAVA_OPTS=""\
    JAR_FILE=${JAR_FILE}

# set time zone
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime & amp; & amp; echo 'Asia/Shanghai' >/etc/timezone
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories \
     & amp; & amp; apk add --no-cache curl
#Copy the jar package in the maven directory to docker and name it for_docker.jar
COPY target/$JAR_FILE $WORK_PATH/


#Set working directory
WORKDIR $WORK_PATH


# Specify the port for external interaction
EXPOSE $EXPOSE_PORT
# Configure the container to make it executable
ENTRYPOINT exec java $JAVA_OPTS -jar $JAR_FILE

k8s deployment template deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {APP_NAME}
  labels:
    app: {APP_NAME}
spec:
  selector:
    matchLabels:
      app: {APP_NAME}
  replicas: {REPLICAS} # Number of Pod replicas
  strategy:
    type: RollingUpdate # rolling update strategy
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      name: {APP_NAME}
      labels:
        app: {APP_NAME}
      annotations:
        timestamp: {TIMESTAMP}
        prometheus.io/port: "50000" # cannot be dynamically assigned
        prometheus.io/path: /actuator/prometheus
        prometheus.io/scrape: "true" # Pod-based service discovery
    spec:
      affinity: # Set the scheduling strategy and adopt multi-host/multi-availability zone deployment
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - {APP_NAME}
              topologyKey: "kubernetes.io/hostname" # The multi-availability zone is "topology.kubernetes.io/zone"
      terminationGracePeriodSeconds: 30 # Graceful termination grace period
      containers:
      - name: {APP_NAME}
        image: {IMAGE_URL}
        imagePullPolicy: Always
        ports:
        - containerPort: {APP_PORT}
        - name: management-port
          containerPort: 50000 # application management port
        readinessProbe: # readiness probe
          httpGet:
            path: /actuator/health/readiness
            port: management-port
          initialDelaySeconds: 30 # delay loading time
          periodSeconds: 10 # retry interval
          timeoutSeconds: 1 # timeout setting
          successThreshold: 1 # health threshold
          failureThreshold: 9 # unhealthy threshold
        livenessProbe: # liveness probe
          httpGet:
            path: /actuator/health/liveness
            port: management-port
          initialDelaySeconds: 30 # delay loading time
          periodSeconds: 10 # retry interval
          timeoutSeconds: 1 # timeout setting
          successThreshold: 1 # health threshold
          failureThreshold: 6 # unhealthy threshold
        resources: # container resource management
          limits: # resource limits (monitor usage)
            cpu: 0.5
            memory: 1Gi
          requests: # Minimum available resources (flexible scheduling)
            cpu: 0.1
            memory: 200Mi
        env:
          - name: TZ
            value: Asia/Shanghai
---
kind: HorizontalPodAutoscaler # elastic scaling controller
apiVersion: autoscaling/v2beta2
metadata:
  name: {APP_NAME}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {APP_NAME}
  minReplicas: {REPLICAS} # scaling range
  maxReplicas: 6
  metrics:
    -type: Resource
      resource:
        name: cpu # Specify the resource index
        target:
          type: Utilization
          averageUtilization: 50

——

We have created a high-quality technical exchange group. When you are with excellent people, you will become excellent yourself. Hurry up and click to join the group and enjoy the joy of growing together. In addition, if you want to change jobs recently, I spent 2 weeks a year ago collecting a wave of face-to-face experience from big factories. If you plan to change jobs after the festival, you can click here to claim it!

Recommended reading

  • Spring Boot 3.1.0 released, adding a lot of new features and improvements

  • ChatGPT official APP is online: extremely fast and free, with added voice recognition!

  • The most comprehensive interpretation of the first batch of 70 plug-ins in ChatGPT Plus

·····················································

Hello, I am DD, a programmer. I have been developing a veteran driver for 10 years, MVP of Alibaba Cloud, TVP of Tencent Cloud. From general development to architect to partner. Along the way, my deepest feeling is that we must keep learning and pay attention to the frontier. As long as you can persevere, think more, complain less, and work hard, it will be easy to overtake on curves! So don’t ask me if it’s too late to do what I do now. If you are optimistic about something, you must persevere to see hope, not to persevere only when you see hope. Believe me, as long as you stick to it, you will be better than now! If you have no direction yet, you can follow me first, and I will often share some cutting-edge information here to help you accumulate capital for cornering and overtaking.

The knowledge points of the article match the official knowledge files, and you can further learn relevant knowledge Java skill treeHomepageOverview 118695 people are studying systematically